Creating custom CPPUnit suite from a file

Hi there Liz,
     I wanted to do the same thing: build a dynamic test suite by reading it from an .ini file, but on the way of getting there stumbled onto, more or less, the "ownership" problem.
     In my case I build 2 test suites:the first contains all existing tests, the second contains only the tests that I select from the first suite; and I run only the second suite. The problem is that I cannot "clean up" - both suites will try do delete the same pointers and this will crash the program.
   
     Did you have any luck in solving your problem?

Regards,
     Dragos

P.S.
     The mfc test runner seems to provide the functionality of creating a dynamic test - or at least running just a particular test from a suite - I have just began analyzing it - maybe we can find a solution inside it.

Hi Liz and Dragos,

I think I have solved your problem... at least for my project (and hopefully for my collegues at LKEB <www.lkeb.nl> as well).  I wrote a TestSelection class, having two data members: a pointer to a Test created by TestFactoryRegistry, and a TestSuite containing /references/ to the selected Test objects.  So this TestSuite data member does /not/ own the tests created by the registry!

Each reference to a selected Test is held by another class that I wrote: TestReference. This class is derived from Test; it forwards each public virtual function from its base class to its data member, the reference.  Except for the destructor!

The TestSuite of my TestSelection class "owns" the TestReference objects, but when a TestReference is destructed, it won't destruct the originally selected Test object.

TestSelection reads the selection from an XML file. I think XML is preferable to a simple plain text file, because of the hierarchical nature of CppUnit's Test class: a Test may own one or more other Test objects, that all may own multiple Test objects as well, recursively. Each XML tag <Test> has one attribute: the name of the test it represents.

Please let me know if you're still interested...

PS The TestSelection class was developed with help from my collegues at LKEB, as well as from members of the ACCU mailing list.

Hi Dragos,

Thanks for being interested  :-)  The TestReference class that I wrote is very simple. Still I think it would be useful to have it added to CppUnit.  Here's approximately what it looks like:

  class TestReference: public Test
  {
  public:
    // Constructor.
    explicit TestReference(Test& arg)
    :
    m_test(arg) {}

    // Destructor does NOT destruct the Test object it referenced.
    ~TestReference() {}

  private:

    // Its only data member.
    Test& m_test;

    // Overriding all virtual functions of Test:

    void run(TestResult* result)
    {
      m_test.run(result);
    }

    int countTestCases() const
    {
      return m_test.countTestCases();
    }

    int getChildTestCount() const
    {
      return m_test.getChildTestCount();
    }

    Test* getChildTestAt(int index) const
    {
      return m_test.getChildTestAt(index);
    }

    std::string getName() const
    {
      return m_test.getName();
    }

    bool findTestPath(const std::string& name, TestPath& path) const
    {
      return m_test.findTestPath(name, path);
    }

    bool findTestPath(const Test* test, CppUnit::TestPath& path) const
    {
      return m_test.findTestPath(test, path);
    }

    Test* findTest( const std::string& name ) const
    {
      return m_test.findTest(name);
    }

    TestPath resolveTestPath(const std::string& path) const
    {
      return m_test.resolveTestPath(path);
    }
   
    Test* doGetChildTestAt(int) const
    {
      CPPUNIT_FAIL("Cannot access protected member of m_test!");
      return 0;
    }
   
    void checkIsValidIndex(int)
    {
      CPPUNIT_FAIL("Cannot access protected member of m_test!");
    }
  };

Is it clear to you?

Kind regards,
--
Niels Dekker
http://www.xs4all.nl/~nd/dekkerware
Scientific programmer at LKEB, Leiden University Medical Center

Note that the TestReference class I just posted looks a bit similar to CppUnit::TestDecorator.  But TestDecorator does actually delete the m_test it holds, so it can't be used to solve our "ownership problem".

The documentation of CppUnit::TestDecorator suggests otherwise, but I think it's outdated. So I just reported:
  "[ 2185407 ] TestDecorator does assume ownership!"
http://sourceforge.net/forum/forum.php?thread_id=1880533&forum_id=37108

Oops, the correct link to my little report ("TestDecorator does assume ownership!")is:
https://sourceforge.net/tracker/index.php?func=detail&aid=2185407&group_id=11795&atid=111795

Lizozom wrote on 2007-11-28:
> I would like to allow a user to run a list of tests
> from a text file that looks like this:
> 
> TestFixture1::TestMethod1
> TestFixture1::TestMethod2
> TestFixture2::TestMethod2

When using the TestReference class that I posted here before (2008-10-18), you can write a function to add references of the tests listed in such a text file to your custom test suite, as follows:

void addTestsFromTextFileToTestSuite(
    Test& allTests, // All tests from the registry.
    const std::string& fileName, // Input file name.
    TestSuite& testSuite) // Custom test suite.
{
    std::ifstream fileStream( fileName.c_str() );
    std::string testName;

    while ( getline(fileStream, testName) )
    {
        Test* foundTest = allTests.findTest(testName);
        testSuite.addTest(new TestReference(*foundTest));
    }
}

Note that the foundTest objects are owned by allTests, while the created TestReference objects are owned by testSuite. This works well because TestReference does not delete foundTest.

So do you think I've solved your problem?
Anyway, I still prefer to have the names of the selected tests in an XML file...

Hi Niels,
   
    I had some yesterday to look at the problem. addTestsFromTextFileToTestSuite is working just great (using the TestReference class). Thanks.

   Another solution would be to have a ModifiedTestSuite that is exactly the same as TestSuite except in one aspect : will not delete the tests in the suite

void
ModifiedTestSuite::deleteContents()
{
  //int childCount = getChildTestCount();
  //for ( int index =0; index < childCount; ++index )
  //  delete getChildTestAt( index );

  m_tests.clear();
}

   Using this class addTestsFromTextFileToTestSuite would become

void addTestsFromTextFileToTestSuite(
Test& allTests, // All tests from the registry.
const std::string& fileName, // Input file name.
ModifiedTestSuite& testSuite) // Custom test suite.
{
std::ifstream fileStream( fileName.c_str() );
std::string testName;

while ( getline(fileStream, testName) )
{
Test* foundTest = allTests.findTest(testName);
modifiedTestSuite.addTest(foundTest));
}
}

  - I tested it and it works - no memory leaks. What's your opinion on this approach?

Hi all,
    I have now a first version of the program that parses an XML file and then creates a custom test suite (also includes generating the xml) - let me know if you are interested.

[long message warning :) ]

The XML structure will be something this:

<?xml version="1.0"?>
<TestConfiguration>
   
    <Registry name="FirstRegistry" testSuiteName="TestSuiteName">
        <test name="Test1" addToRunner="true"/>
        <test name="Test2" addToRunner="true"/>
        <test name="Test3" addToRunner="true"/>
       
    </Registry>
   
    <Registry name="SecondRegistry" testSuiteName="TestSuiteName">
        <test name="Test1" addToRunner="true"/>
        <test name="Test2" addToRunner="true"/>
        <test name="Test3" addToRunner="true"/>
       
    </Registry>
   
    ...
</TestConfiguration>

If we modify the addToRunner attribute to something else than "true" the test will not be executed

Now for the code: ( I stripped down to code to the important parts)

#define CHECKHR(hr)\ {\     if( hr != S_OK)\     return false;\ }

bool prepareSuiteToRun(/*in*/CString& pathToXMLFile,/*out*/TestSuite*& allTestSuite,/*out*/ ModifiedTestSuite*& suiteToRun)
{
    CComPtr<IXMLDOMDocument> pXMLDoc;
    HRESULT hr = S_FALSE;
    hr = pXMLDoc.CoCreateInstance(CLSID_DOMDocument);
    CHECKHR(hr);
    VARIANT_BOOL bSuccess = false;
    CComVariant nameOfXML(pathToXMLFile);
    hr = pXMLDoc->load(nameOfXML,&bSuccess);
    CHECKHR(hr);
   
    /*
     *    position ourselves to the TestConfiguration node
     */
    CComPtr<IXMLDOMNode> pXMLTestConfigurationNode;
    hr = pXMLDoc->selectSingleNode(CComBSTR("TestConfiguration"),&pXMLTestConfigurationNode);
    CHECKHR(hr);
   
    /*
     *    iterate through the registries
     */
    CComPtr<IXMLDOMNodeList> pXMLListOfRegistries;
    hr = pXMLTestConfigurationNode->get_childNodes(&pXMLListOfRegistries);
    CHECKHR(hr);

    long nrOfRegistries = -1;
    hr = pXMLListOfRegistries->get_length(&nrOfRegistries);
    CHECKHR(hr);
    if(nrOfRegistries <= 0)
    {
        return false;
    }

    /*
     *    for each registry
     */
    for(int regIndex = 0 ; regIndex < nrOfRegistries; ++regIndex)
    {
        CComPtr<IXMLDOMNode> registry;
        hr = pXMLListOfRegistries->get_item(regIndex,&registry);
        CHECKHR(hr);
        CComPtr<IXMLDOMNamedNodeMap> registryAttributes;
        hr = registry->get_attributes(&registryAttributes);
        CHECKHR(hr);
        CComPtr<IXMLDOMNode> registryNameNode;
        hr = registryAttributes->getNamedItem(CComBSTR("name"),&registryNameNode);
        CHECKHR(hr);
        CComVariant registryName;
        hr = registryNameNode->get_nodeValue(&registryName);
        CHECKHR(hr);
        CppUnit::TestFactoryRegistry::getRegistry(string(CStringA(registryName))).addTestToSuite(allTestSuite);

        /*
         *    we only have one suite per registry
         */
        CComPtr<IXMLDOMNode> testSuiteNameNode;
        hr = registryAttributes->getNamedItem(CComBSTR("testSuiteName"),&testSuiteNameNode);
        CHECKHR(hr);
        CComVariant testSuiteName;
        hr = testSuiteNameNode->get_nodeValue(&testSuiteName);

        CComPtr<IXMLDOMNodeList> pXMLListOfTests;
        hr = registry->get_childNodes(&pXMLListOfTests);
        CHECKHR(hr);
       
        long nrOfTests = -1;
        hr = pXMLListOfTests->get_length(&nrOfTests);
        CHECKHR(hr);
        if(nrOfTests > 0)
        {
            /*
             *    for each test in the suite
             */
            for (int testIndex = 0; testIndex < nrOfTests ; ++testIndex)
            {
                CComPtr<IXMLDOMNode> testNode;
                hr = pXMLListOfTests->get_item(testIndex,&testNode);
                CHECKHR(hr);
                CComPtr<IXMLDOMNamedNodeMap> testAttributes;
                hr = testNode->get_attributes(&testAttributes);
                CHECKHR(hr);
                CComPtr<IXMLDOMNode> testNameNode;
                hr = testAttributes->getNamedItem(CComBSTR("name"),&testNameNode);
                CHECKHR(hr);
                CComVariant testName;
                hr = testNameNode->get_nodeValue(&testName);
                CHECKHR(hr);

                CComPtr<IXMLDOMNode> addToRunnerNode;
                hr = testAttributes->getNamedItem(CComBSTR("addToRunner"),&addToRunnerNode);
                CHECKHR(hr);
                CComVariant addToRunnerValue;
                hr = addToRunnerNode->get_nodeValue(&addToRunnerValue);
                CHECKHR(hr);

                CStringA addIfTrue(addToRunnerValue);
                if(addIfTrue.CompareNoCase("true") == 0)
                {
                    CStringA completeTestName(testSuiteName);
                    completeTestName = completeTestName + "::" + CStringA(testName);
                    string testNameString(completeTestName);
                    Test* foundTest = allTestSuite->findTest(testNameString);
                    if(foundTest)
                    {
                        suiteToRun->addTest(foundTest);
                    }
                }
            }
        }
    }
    return true;
}

void AddWhiteSpaceToNode( CComPtr<IXMLDOMDocument>& pDom,CComBSTR bstrWs,CComPtr<IXMLDOMElement>& pNode )
{
    HRESULT hr;
    CComPtr<IXMLDOMText> pws = 0;
    CComPtr<IXMLDOMNode> pBuf = 0;
    hr = pDom->createTextNode(bstrWs,&pws);
    hr = pNode->appendChild(pws,&pBuf);
}

bool createXML(const CString& pathToXML,const vector<string> listOfRegistries)
{
    CComPtr<IXMLDOMDocument> pXMLDoc;
    HRESULT hr = S_FALSE;
    hr = pXMLDoc.CoCreateInstance(CLSID_DOMDocument);
    CHECKHR(hr);
 
    //the processing instructions <?xml version="1.0"?>
    CComPtr<IXMLDOMProcessingInstruction> pi= 0;
    CComBSTR xml("xml");
    CComBSTR xmlVersion("version = '1.0'");
    hr = pXMLDoc->createProcessingInstruction(xml,xmlVersion,&pi);
    CComPtr<IXMLDOMNode> pTmpNode = 0;
    hr =  pXMLDoc->appendChild(pi,&pTmpNode);
    if(pTmpNode)
    {
        pTmpNode.Release();
        pTmpNode = 0;
    }
    CComBSTR bstr_wsn(L"\n");
    CComBSTR bstr_wsnt(L"\n\t");
    CComBSTR bstr_wsntt(L"\n\t\t");

    //the TestConfiguration category
    CComPtr<IXMLDOMElement> pCategoryInfo = 0;
    hr = pXMLDoc->createElement(CComBSTR("TestConfiguration"),&pCategoryInfo);
    hr = pXMLDoc->appendChild(pCategoryInfo,&pTmpNode);
    if(pTmpNode)
    {
        pTmpNode.Release();
        pTmpNode = 0;
    }

    //add new line and tab for formatting
    AddWhiteSpaceToNode(pXMLDoc,bstr_wsnt,pCategoryInfo);

    for (unsigned int indexOfRegistry = 0 ; indexOfRegistry < listOfRegistries.size(); ++indexOfRegistry )
    {
        AddWhiteSpaceToNode(pXMLDoc,bstr_wsnt,pCategoryInfo);
        TestSuite* temporarySuite = new CppUnit::TestSuite("Temporary Suite");
        CppUnit::TestFactoryRegistry::getRegistry(listOfRegistries[indexOfRegistry]).addTestToSuite(temporarySuite);
        CComPtr<IXMLDOMElement> pRegistryInfo;
        hr = pXMLDoc->createElement(CComBSTR("Registry"),&pRegistryInfo);
        CHECKHR(hr);
        pRegistryInfo->setAttribute(CComBSTR("name"),CComVariant(listOfRegistries[indexOfRegistry].c_str()));
        //get suite name
        if(temporarySuite->getChildTestCount() !=1)
        {
            return false;
        }
        string suiteName = temporarySuite->getChildTestAt(0)->getName();
        pRegistryInfo->setAttribute(CComBSTR("testSuiteName"),CComVariant((temporarySuite->getChildTestAt(0)->getName()).c_str()));
        pCategoryInfo->appendChild(pRegistryInfo,&pTmpNode);
        if(pTmpNode)
        {
            pTmpNode.Release();
            pTmpNode = 0;
        }

       for(int nrOfTests = 0; nrOfTests < temporarySuite->getChildTestAt(0)->getChildTestCount(); ++nrOfTests)
       {
         //indentation for better formatting
            AddWhiteSpaceToNode(pXMLDoc,bstr_wsntt,pRegistryInfo);
           CComPtr<IXMLDOMElement> pTestInfo;
           hr = pXMLDoc->createElement(CComBSTR("test"),&pTestInfo);
           CHECKHR(hr);
           CStringA testNameString = (temporarySuite->getChildTestAt(0)->getChildTestAt(nrOfTests)->getName()).c_str();
           int tokenPos = testNameString.Find("::");
           pTestInfo->setAttribute(CComBSTR("name"),CComVariant(testNameString.Right(testNameString.GetLength() - tokenPos - 2)));
           pTestInfo->setAttribute(CComBSTR("addToRunner"),CComVariant("true"));
           pRegistryInfo->appendChild(pTestInfo,&pTmpNode);
           if(pTmpNode)
           {
               pTmpNode.Release();
               pTmpNode = 0;
           }
       }
       
        AddWhiteSpaceToNode(pXMLDoc,bstr_wsnt,pRegistryInfo);
        AddWhiteSpaceToNode(pXMLDoc,bstr_wsnt,pCategoryInfo);
        delete temporarySuite;
        temporarySuite = 0;
    }
 
    AddWhiteSpaceToNode(pXMLDoc,bstr_wsn,pCategoryInfo);
    hr = pXMLDoc->save(CComVariant(pathToXML));
    CHECKHR(hr);
    return true;
};

int _tmain(int argc, _TCHAR* argv[])
{
    //with a switch you can use this exe just to generate the xml
   
    ....
    CoInitializeEx(0, COINIT_MULTITHREADED);
    vector<string> listOfRegistries;
    listOfRegistries.push_back(string("FirstModuleTest"));
    listOfRegistries.push_back(string("SecondModuleTest"));
   
    CppUnit::TestSuite *allTestsSuite = new CppUnit::TestSuite("All tests");
    CppUnit::ModifiedTestSuite* suiteToRun = new CppUnit::ModifiedTestSuite( "Tests to run");
    if(!prepareSuiteToRun(pathToXml,allTestsSuite,suiteToRun))
    {
        return -1;
    }
    CppUnit::TextUi::TestRunner runner;
      runner.addTest(suiteToRun );
      runner.setOutputter(new CppUnit::XmlOutputter( &runner.result(),std::cerr));
      bool wasSucessful = runner.run();
   
    if(allTestsSuite)
    {
        delete allTestsSuite;
        allTestsSuite = 0;
    }
    CoUninitialize();
    return wasSucessful ? 0 : 1;
}

    There are some limitations atm (like the fact that only one suite per registry is allowed - and I don't check the default registry - only named ones) specific to the tests I have - but those could be easily solved.

    If you have other questions/suggestions for improvement let me know :)

Cheers,
      Dragos