cppunit-devel

RE: [Cppunit-devel] Parameterizing tests by using XML

[Moran Ben-D. <mbe...@ii...>]

To follow your raising of the overhead issue.. 

Although XML is great, it is a bit beurocratic when all you want to send
your test case is a an integer.. I think the best approach to data driving
the test cases is to start with stl containers (e.g. map).. e.g. a test case
receives a list of keys and values created when the test is setup in the
runner.

As for the design of this, I am not sure.. however, as long as the person(s)
writing and feeding the tests are able to decide data structure that is
passed (e.g. stl container, xml dom, etc.), the framework will be best.

thanks,
moran

-----Original Message-----
From: Duane Murphy [mailto:dua...@ma...]
Sent: Thursday, July 18, 2002 12:10 PM
To: CppUnit Developers
Subject: Re: [Cppunit-devel] Parameterizing tests by using XML


I think moving to or adding a data driven interface to cppUnit would be a
great feature. I have considered it myself.

XML is certainly the right way of doing this. It allows tests to be
written by hand or generated by some automated tool. This certainly
doesnt absolve the developer from writing test code, but a good design
should make it easier to focus on the test rather than the test
environment. One of my personal deterants to cppunit has been the
overhead involved in getting test registered in order to run. A data
driven approach would be great.

[ more comments below ]

--- At Thu, 18 Jul 2002 16:07:43 +0200, Steffen Huebner wrote:

>I have looked over CppUnit the last time to find a way making tests more
>flexible by parameterizing them.
>The idea is to place input parameters and corresponding expected values
>in a XML file. 
>That forces to parse the incoming XML file.
>Having a common interface that gives the possibility to parse the XML
>file is our way to do so.
>At the moment I don`t really know how this interface should look and
>which methods it should provide.
>
>My questions about this are:
>1. What do you think about an interface with the possibility to use
>different parsers?

The XML interface should not depend on any parser. Standard XML features
should work with most any parser. 

I suppose each parser does have its own interface. I was think more along
the lines of the XML file definition.

>2. Does such an interface for parsing already exists?
>
>I found MS XML and XML4C by Alphaworks(IBM) which are both useable in C++.

There is also libxml and libxml++. libxml++ is apparently still under
development. I wonder if an abstract interface can be developed that
would more easily allow different parsers to be used.

>3. Would that be a nice feature for your CppUnit-Framework?

Yes it would.

 ...Duane

-- 
"If tyranny and oppression come to this land, it will be in the
guise of fighting a foreign enemy."              - James Madison




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Cppunit-devel mailing list
Cpp...@li...
https://lists.sourceforge.net/lists/listinfo/cppunit-devel

Re: [Cppunit-devel] Parameterizing tests by using XML

[FUKUDA F. <ff...@nt...>]

Hi.

--- "[Cppunit-devel] Parameterizing tests by using XML" / Steffen Huebner / 2002/07/18 16:07:43 +0200 ---
>I have looked over CppUnit the last time to find a way making tests more 
>flexible by parameterizing them.
>The idea is to place input parameters and corresponding expected values in a XML file. 
>That forces to parse the incoming XML file.
>...

I'm now trying this. 

- format of testdata(XML) is identical to JTestCase (http://jtestcase.sourceforge.net/)

<tests>
  <class name="CalculatorTest">
    <!-- Sample global params --> 
    <params>
      <param name="gVar1" type="double">10</param>
      <param name="gVar2" type="int">30</param>
    </params>
    <!-- params & asserts for CalculatorTest::testCalculate() -->
    <method name="testCalculate" test-case="positive-add">
      <!-- expected: 10 + 20 = 30 -->
      <params>
        <param name="var1" type="int">10</param>
        <param name="var2" type="int">20</param>
        <param name="opt">+</param>
      </params>
      <asserts>
        <assert name="result" type="int" action="EQUALS">30</assert>
      </asserts>
    </method>
    <!--- and so on -->
  </class>
</tests>

... and I made class: TextContext/TestContextProxy.
you can use as follows:

class CalculatorTest : public CppUnit::TestFixture {
  CPPUNIT_TEST_SUITE(CalculatorTest);
  CPPUNIT_TEST(testCalculate);
  CPPUNIT_TEST_SUITE_END();
private:

  TestContext* tc_;

  static int atoi(const std::string& str) {
    int result;
    std::istringstream strm(str);
    strm >> result;
    return result;
  }

public:

  virtual void setUp() {
    // load an XML & set the name of class
    tc_ = new XercesTestContext("../data/testcase.xml", "CalculatorTest");
  }
  virtual void tearDown()
    { delete tc_; }

  void testCalculate() {
    // make a proxy from TestContext & name of method
    TestContextProxy proxy(tc_,"testCalculate");
    // enumerate params/asserts 
    while ( proxy.advance() ) {
      // extract params
      TestContext::params_type params = proxy.getTestCaseParams();
      int x = atoi(params["var1"]);
      int y = atoi(params["var2"]);
      char op = params["opt"][0];
      // execute
      Calculator c;
      int result = c.calculate(x,y,op);
      // verify!
      CPPUNIT_ASSERT_CONTEXTPROXY(proxy, "result", result);
    }
  }
};

sounds good? ;-)

-----:-----:-----:-----:-----:-----:-----:-----:-----:-----
FUKUDA (episteme) Fumiki -- magical, but never a magic...

RE: [Cppunit-devel] Parameterizing tests by using XML

[Michel A. <Arm...@si...>]

Hi all.

Actually I was thinking about a little bit different solution here. It
differs mainly in two points:

1. the framework should be responsible for parsing the xml file and invoke
the test with a particular set of parameters (as opposed to let each of the
test cases perform the parsing)

2. the DTD of the xml file is a less sophisticated

It follows a short description of my suggestion:

A (new) BEGIN_TEST_SUITE macro is able to accept the name of a (input-)
file. If this (new) macro is used a new type of TestCaller
(ParameterizedTestCaller) will be used for this suite. If a
ParameterizedTestCaller  is executing its test, it will scan the in-file for
a section for this particular test. The section will be parsed and a
TestSuite object will be created that contains one Test object for each set
of parameters. This newly created TestSuite object will then be executed
using the run(TestResult* result) method.
The (parameterized) test method will be called with two std::istreams: the
one contains the (input) parameters, the other one contains  expected
results (can also be an empty stream). Thus the test method is able to read
its parameters using the operator>>. If there is no section for a test
method, it is only called once (with no parameters?).

Example:

MyTest.xml:

<test name=”testMethod”>
    <call name=”simple”>
        <description>
            Simple and valid set of parameters.
            Format of “in” is: “int int std::string”
            Format of “exp” is: “int”
        </description>
        <in>
            3 -1 myCorrectString
        </in>
        <exp>
            25
        </exp>
    </call>
</test>

MyTest.cpp:

void testMethod(std::istream& param_in, std::istream& exp_in)
{
    int param1, param2;
    std::string param3;

    param_in >> param1 >> param2 >> param3;

    int exp;
    
    exp_in >> exp;

    // perform tests using these parameters ...
}

I was also thinking about introducing a TestContext class that would be
responsible for hosting the information gained from the xml file (e.g. input
and expected streams). But I am not sure about how to do this in a way that
a improved flexibility/exchangability is gained. I will expand this issue in
a separate email...

Regards

Armin


> -----Original Message-----
> From: FUKUDA Fumiki [mailto:ff...@nt...]
> Sent: Freitag, 19. Juli 2002 01:25
> To: cpp...@li...
> Subject: Re: [Cppunit-devel] Parameterizing tests by using XML
> 
> 
> Hi.
> 
> --- "[Cppunit-devel] Parameterizing tests by using XML" / 
> Steffen Huebner / 2002/07/18 16:07:43 +0200 ---
> >I have looked over CppUnit the last time to find a way 
> making tests more 
> >flexible by parameterizing them.
> >The idea is to place input parameters and corresponding 
> expected values in a XML file. 
> >That forces to parse the incoming XML file.
> >...
> 
> I'm now trying this. 
> 
> - format of testdata(XML) is identical to JTestCase 
> (http://jtestcase.sourceforge.net/)
> 
> <tests>
>   <class name="CalculatorTest">
>     <!-- Sample global params --> 
>     <params>
>       <param name="gVar1" type="double">10</param>
>       <param name="gVar2" type="int">30</param>
>     </params>
>     <!-- params & asserts for CalculatorTest::testCalculate() -->
>     <method name="testCalculate" test-case="positive-add">
>       <!-- expected: 10 + 20 = 30 -->
>       <params>
>         <param name="var1" type="int">10</param>
>         <param name="var2" type="int">20</param>
>         <param name="opt">+</param>
>       </params>
>       <asserts>
>         <assert name="result" type="int" action="EQUALS">30</assert>
>       </asserts>
>     </method>
>     <!--- and so on -->
>   </class>
> </tests>
> 
> ... and I made class: TextContext/TestContextProxy.
> you can use as follows:
> 
> class CalculatorTest : public CppUnit::TestFixture {
>   CPPUNIT_TEST_SUITE(CalculatorTest);
>   CPPUNIT_TEST(testCalculate);
>   CPPUNIT_TEST_SUITE_END();
> private:
> 
>   TestContext* tc_;
> 
>   static int atoi(const std::string& str) {
>     int result;
>     std::istringstream strm(str);
>     strm >> result;
>     return result;
>   }
> 
> public:
> 
>   virtual void setUp() {
>     // load an XML & set the name of class
>     tc_ = new XercesTestContext("../data/testcase.xml", 
> "CalculatorTest");
>   }
>   virtual void tearDown()
>     { delete tc_; }
> 
>   void testCalculate() {
>     // make a proxy from TestContext & name of method
>     TestContextProxy proxy(tc_,"testCalculate");
>     // enumerate params/asserts 
>     while ( proxy.advance() ) {
>       // extract params
>       TestContext::params_type params = proxy.getTestCaseParams();
>       int x = atoi(params["var1"]);
>       int y = atoi(params["var2"]);
>       char op = params["opt"][0];
>       // execute
>       Calculator c;
>       int result = c.calculate(x,y,op);
>       // verify!
>       CPPUNIT_ASSERT_CONTEXTPROXY(proxy, "result", result);
>     }
>   }
> };
> 
> sounds good? ;-)
> 
> -----:-----:-----:-----:-----:-----:-----:-----:-----:-----
> FUKUDA (episteme) Fumiki -- magical, but never a magic...
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Cppunit-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppunit-devel
> 

Re: [Cppunit-devel] Parameterizing tests by using XML

[Robert W. <ro...@po...>]

Hi,

On Friday 19 July 2002 08:36, Michel Armin wrote:
> Hi all.
>
> Actually I was thinking about a little bit different solution here.
> It differs mainly in two points:
>
> 1. the framework should be responsible for parsing the xml file and
> invoke the test with a particular set of parameters (as opposed to
> let each of the test cases perform the parsing)
>
> 2. the DTD of the xml file is a less sophisticated

These are _very_ important points -- you would not want to need test 
cases for your test cases, right?

I still see two big drawbacks on using a XML driven test.

First it drops the advantage of having the tests written in the same 
language as the code under test. Developers have to learn another new 
language / syntax / semantics and can not focus on their real work. 
Even though XML looks easy, it has some pitfalls you will most likely 
uncover and which will cost really lots of time, nerves, and sanity.

Second I don't think the XML overhead justifies the results. How many 
tests do you have that run the same way that you just need to exchange 
the input and output values? (And how many of those use primitive 
inputs and outputs?) If I ever have to write more than three tests for 
the same method, I think hard about abetter design, usually the method 
has just too much responsibilities.

Just my 2 cents,

Robert

RE: [Cppunit-devel] Parameterizing tests by using XML

[Michel A. <Arm...@si...>]

> -----Original Message-----
> From: Robert Wenner [mailto:ro...@po...]
> Sent: Freitag, 19. Juli 2002 09:01
> To: Michel Armin; cpp...@li...
> Subject: Re: [Cppunit-devel] Parameterizing tests by using XML
> 
> 
> Hi,
> 
> On Friday 19 July 2002 08:36, Michel Armin wrote:
> > Hi all.
> >
> > Actually I was thinking about a little bit different solution here.
> > It differs mainly in two points:
> >
> > 1. the framework should be responsible for parsing the xml file and
> > invoke the test with a particular set of parameters (as opposed to
> > let each of the test cases perform the parsing)
> >
> > 2. the DTD of the xml file is a less sophisticated
> 
> These are _very_ important points -- you would not want to need test 
> cases for your test cases, right?
> 
> I still see two big drawbacks on using a XML driven test.
> 
> First it drops the advantage of having the tests written in the same 
> language as the code under test. Developers have to learn another new 
> language / syntax / semantics and can not focus on their real work. 
> Even though XML looks easy, it has some pitfalls you will most likely 
> uncover and which will cost really lots of time, nerves, and sanity.

That's of course true, but I do not want to get rid of the current flavor of
programming tests; I just want to add a new possibility to write tests.

> Second I don't think the XML overhead justifies the results. How many 
> tests do you have that run the same way that you just need to 
> exchange 
> the input and output values? (And how many of those use primitive 
> inputs and outputs?) If I ever have to write more than three 
> tests for 
> the same method, I think hard about abetter design, usually 
> the method 
> has just too much responsibilities.

Actually almost all of my tests would benifit from such a parameterized
testing approach. (Most of our code is concerned with manipulating images.
Of course, we support a number of different types of images. Our functions
should be able to deal with all of these types of images in a transparent
way. If I have a possibility to parameterize the type of the image used for
a test, I could greatly improve the test coverage.)

Armin.

Re: [Cppunit-devel] Parameterizing tests by using XML

[Robert W. <ro...@po...>]

On Friday 19 July 2002 09:19, Michel Armin wrote:
> Actually almost all of my tests would benifit from such a
> parameterized testing approach. (Most of our code is concerned with
> manipulating images. Of course, we support a number of different
> types of images. Our functions should be able to deal with all of
> these types of images in a transparent way. If I have a possibility
> to parameterize the type of the image used for a test, I could
> greatly improve the test coverage.)

Wouldn't a template be more helpful?
Or a (abstract) base class that excercises test for all images and is 
not directly called but its derived classes call it and offer 
specialised methods the base calls (template class pattern)?

Robert

RE: [Cppunit-devel] Parameterizing tests by using XML

[Moran Ben-D. <mbe...@ii...>]

I'd just like to add to the discussion with a point about usefullness of
parameterizing tests..

The protocol for passing parameters to tests cases that I'm actually looking
for is one which allows my command line tools to pass in variables to my
tests.. that is.. I always create a command line tool for running the tests
for the case where i need to check the validity of component on a remote
system for support, etc.. thus, my tests have several contexts in which they
run:

	1. development (my machine)
	2. qa (centralized server)
	3. other developers machines
	4. production (which can often mean two machines if there is a
staging server)

Just thought I throw this into the salad of parameterizing tests.. XML in my
case is too beaurocratic to run in the case of the QA or production
servers.. i need a quick way of running a parameterized test and XML as
simple as it is, still ends up costing too much effort on my part (e.g. with
issues like is the installed xml parser version correct, does it work
properly, etc.)

moran

-----Original Message-----
From: Robert Wenner [mailto:ro...@po...]
Sent: Friday, July 19, 2002 3:01 AM
To: Michel Armin; cpp...@li...
Subject: Re: [Cppunit-devel] Parameterizing tests by using XML


Hi,

On Friday 19 July 2002 08:36, Michel Armin wrote:
> Hi all.
>
> Actually I was thinking about a little bit different solution here.
> It differs mainly in two points:
>
> 1. the framework should be responsible for parsing the xml file and
> invoke the test with a particular set of parameters (as opposed to
> let each of the test cases perform the parsing)
>
> 2. the DTD of the xml file is a less sophisticated

These are _very_ important points -- you would not want to need test 
cases for your test cases, right?

I still see two big drawbacks on using a XML driven test.

First it drops the advantage of having the tests written in the same 
language as the code under test. Developers have to learn another new 
language / syntax / semantics and can not focus on their real work. 
Even though XML looks easy, it has some pitfalls you will most likely 
uncover and which will cost really lots of time, nerves, and sanity.

Second I don't think the XML overhead justifies the results. How many 
tests do you have that run the same way that you just need to exchange 
the input and output values? (And how many of those use primitive 
inputs and outputs?) If I ever have to write more than three tests for 
the same method, I think hard about abetter design, usually the method 
has just too much responsibilities.

Just my 2 cents,

Robert


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Cppunit-devel mailing list
Cpp...@li...
https://lists.sourceforge.net/lists/listinfo/cppunit-devel