[Cppunit-cvs] cppunit2/doc builddoc.bat,NONE,1.1 cppunit2todo.txt,NONE,1.1 default.css,NONE,1.1 feat
Brought to you by:
blep
From: Baptiste L. <bl...@us...> - 2004-11-15 20:41:22
|
Update of /cvsroot/cppunit/cppunit2/doc In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv22756 Added Files: builddoc.bat cppunit2todo.txt default.css features.txt index.txt opentest.txt opentesttodo.txt Log Message: * added some documentations: features, open test, todo for cppunit 2 and open test. docutils is needed to build the doc. --- NEW FILE: opentest.txt --- ====================== Open Test presentation ====================== :Author: Baptiste Lepilleur :Contact: bl...@us... .. contents:: What is Open Test? ================== Open Test provides a mean to easily integrate a testing framework with Open Test `test drivers`__. __ TestDriver_ The open test concept emerged from the complete separation of the way tests of running test from driving and monitoring the test run in CppUnit 2. Those two activities are managed by two components: the TestRunner_ and the TestDriver_. While implementing a TestRunner_ is cheap, implementing TestDriver_ cost a lot, especially if the test driver must provide a graphical user interface. With Open Test, it is possible to reuse the existing TestDriver_ and implement a TestRunner_ for a specific test framework. This makes implementing a specific functional test framework cheap since you only need to implement the testing part. All the monitoring, user interface parts are already implemented and reusable. .. _TestRunner: What is a TestRunner? ===================== A test runner is a lightweight component that provides the following services: * declare all the available tests * run a specific list of tests and track the run of each test. [Notes: need to be enhanced, resource acquisition service...] .. _TestDriver: What is a TestDriver? ===================== A test driver as many responsibility: * monitoring the test run and detecting crash and infinite-loop to resume the test run. * selecting the test(s) to run * providing feedback during the test run (what test is being run, did the test pass...) * generating report of the test run (output result in XML, compiler and human friendly text format...) A test driver can usually be configured from both the command-line and a configuration file for automated testing. Graphical test drivers are provided for interactive browsing of the test hierarchy, and visual feedback... It should be noted that some test drivers could be connected to multiple test runner at the same time. How does the TestDriver communicate with the TestRunner? ======================================================== While it is possible for a TestDriver_ to directly call methods of a TestRunner_, it is not the way it is usually done. Implementing reliable test run monitoring requires running the test in a separate process. Instead of going the easy way and using shared memory to share test data structure, communication between the TestDriver_ and a TestRunner_ rely on a duplex stream communication. While a little more complex to implement, this have the great advantage that a TestDriver_ can be connected to any TestRunner_ that implements the same protocol. A TestDriver_ can communicate with a TestRunner_ that: * is not using the same run-time version of the C library (a common issue on the Windows platform) * is not compiled using the same compiler * is running on a different architecture (is a duplex network connection is used) * is possibly implemented using another language (would require porting the communication layer to the alternate language) .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: --- NEW FILE: opentesttodo.txt --- ============== Open Test todo ============== :Author: Baptiste Lepilleur :Contact: bl...@us... .. contents:: .. _TestDriver: opentest.html .. _TestRunner: opentest.html .. _`Open Test`: opentest.html General status ============== `Open Test`_ is a fairly new concept and hence has a low level of maturity. Everything is to be done. The TestRunner_ interface has been defined and implemented for CppUnit 2. Everything related to test driver need to be defined and implemented. This involves: * defining property tree for suite and test declaration, test runner configuration and test result. * writing a simple text based TestDriver_ * test monitoring and duplex communication through stream * test driver xml output * extend the text based TestDriver_ to accept multiple TestRunner * GUI based TestDriver_ * allow TestDriver selection from command-line Property tree structure ======================= Suite and test declarations --------------------------- TestDeclarator::beginSuite() and TestDeclarator::addTest(). * time-out delay * description, * input/expected values (for functional testing based on input/output) Test runner configuration ------------------------- TestRunner::runTests(). * is logging enabled * number of threads Test configuration ------------------ TestPlanEntry::configuration(). Two categories of data are associated to a test at this stage of the testing: * test specific runner configuration * test descriptive data Test specific runner configuration: * enable logging Test descriptive data: * input/expected values override Test execution result --------------------- TestRunTracker::addTestInfo(). Notes: this method probably needs to be modified since it can be called multiple times (for example, every time a line of log is added). * test status: success ? * test status type: success/failure/fault... * test result message * test result source code location * test result (input/) actual/expected * test result log .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: Simple text based TestDriver ============================ Implementing a simple text based TestDriver is required to validate the interface of the TestRunner. It should use direct method call for communication with the TestRunner (which means no monitoring). Features: * text progress (like CppUnit BriefProgressListener) * dynamic log output as the test runner * compiler output (assertion location in a compiler compatible format with failure detail below). * optionally select test to run from the command-line Status: partially implemented in opentest/texttestdriver.h. Test monitoring =============== The test monitoring feature needs to be implemented. It requires that the TestDriver_ spawn a child process that run the TestRunner_ and communicate with it. Communication between the TestDriver_ and the TestRunner_ should be stream based as it allows the usage of various implementation, such as shared memory, pipe, and network... Ideally, all implementations should provide the same interface, only requiring the TestDriver to bother about the implementation type at connection time. A spike solution is being worked on to explore shared memory on Windows (Baptiste: a working implementation is in place but need a lot of clean-up). GUI based TestDriver ==================== * should provide all the functionalities of a text based TestDriver (it simply does not close the result window when the test are done). * should support for multiple TestRunner * allow loading of multiple test plug-in (spawn test plug-in wrapper application for each DLL) QT will likely be the first GUI toolkit to be ported as it is multi-platform and is easier to implement than MFC or wxWidget. TestDriver selection from command-line ====================================== It should be possible to select the TestDriver to use from the command-line. This implies that all test drivers provide a similar interface. The user should be able to choose between text driver and GUI driver from the command-line (use plug-in for TestDriver ?). --- NEW FILE: features.txt --- =============================== CppUnit 2 features presentation =============================== :Author: Baptiste Lepilleur :Contact: bl...@us... .. _`Open Test`: opentest.html .. contents:: Supports for quick develop/test cycle ===================================== The framework help having a quick develop/test cycle with the following features: * test factory registry to reduce build time (only the modified test source file need to be recompiled) * compiler format source location of failed assertion to jump straight to the assertion in the source code * detailed diagnostic on assertion failure, frequently avoiding the need to run the debugger * test plug-in to reduce link-time on large project In a well-tuned environment that cycle can be as short as a few seconds, as you usually only recompile only two translation units (the test and the tested implementation), and run the test automatically after a successful build. This makes test driven design possible in C++. Component based testing with test plug-in ========================================= Tests can be embedded into the dynamic library to test. An application provided with the framework is used to run the test contained in a dynamic library. The application can run the test of a specific dynamic library or of multiple dynamic libraries at once. Test-plug in also help separating the tests from how they are run, thus allowing to centralize the way tests are run and their result reported. Test monitor to detect crash and infinite loop ============================================== The framework provides easy mechanism to monitor the test run. It can automatically spawn a child process that run the test and monitor it for crash or infinite loop. In the case of infinite loop it terminates the child process. If either a crash or an infinite loop occurred, a new child process is spawned and testing resume after the problematic test case. Graphical test driver ===================== Graphical test driver are provided to: * navigate test hierarchy * run a specific test or test suite * browse test result Open test runner interface ========================== The test drivers (the components that run and monitor the tests) can work with any test runner that implements the `Open Test`_ interface. This means that the framework graphical test driver can be used to run and monitor test implemented using another test framework as long as the `Open Test`_ interface can be implemented for that other framework. More detail can be found in the `Open Test`_ document. Flexible output of test result ============================== The result of a test run is thoroughly inspectable. Outputter are provided to output the result in multiple format, such as XML for integration in a tool chain, or a human readable format with compiler like assertion location for integration with IDE. Extensive test driver configuration =================================== Test drivers can be configured either using the command-line or persistent configuration files. The configuration management is designed in such a way that new configuration options can easily be added. Custom assertion support ======================== The assertion design separate the assertion location tracking from the function call used to implement the assertion. This means that assertion functions can have a variable number of parameters, and that default parameters are allowed. Here is an example of how the assertion ``CPPUT_FAIL( "bad parameters." )`` is expanded. The assertion is defined as follow: :: # define CPPUT_FAIL \ CPPUT_BEGIN_ASSERTION_MACRO() \ ::CppUT::fail Which is expanded into: :: ::CppUT::setCheckPointLocation( ::CppUT::Location( __FILE__, __LINE__ ) ), // notes the comma ::CppUT::fail Hence, the code ``CPPUT_FAIL( "bad parameters." )`` first call ``SetCheckPointLocation`` to memorize where the assertion occurred in the source code, and then call the function ``fail`` with the parameter(s) specified by the user. As the parameters are not part of the macro, their number can vary. One of the advantages of writing custom assertions is that in addition to detecting that the assertion failed, a detailed diagnostic can be provided. For example, when comparing two structured trees, knowing that they are different let you know that something is wrong, but not *what*. With custom assertions, the assertion can be tested, and a detailed diagnostic based on the knowledge of what the structure represent and what would be useful to diagnostic the failure can be reported. An application of advanced assertions can be found in `enumeration and string assertions`_. Non aborting assertions support =============================== The framework provides both assertions that abort the current test (throwing an exception), and assertions that do not abort the current test but keep track of the failures. Those assertions are named respectively "aborting assertions " and "checking assertions". The implementation of this feature relies on a mechanism similar to `setCheckPointLocation()` described in `Custom assertion support`_, meaning that all aborting assertions can easily be made available as checking assertions and vice versa. Property tree as test result for integration ============================================ The result of each test case run is always a property tree. All data related to the execution of a test case are stored as property in this tree. For instance, the status of the test (success, failure...) is just a specific property in this tree. Using a property tree provide a lot of flexibility as it allows association of complex data to each test run such as: * duration of the test * log messages * alternative test execution status (succeeded with warning, failure because of missing resources...) * actual and expected value Being able to report actual and expected values in a test result is a very powerful features. For example, if you are generating input values from a spreadsheet application, after running the test, you can put the actual values back into the spreadsheet. Property tree associated to each test declaration ================================================= Each test has a property tree. The property tree can be used to store descriptive data for each test. Property can be marked as "inheritable", meaning all tests of a test suite inherit this property (but they can override it). Here is some example of properties that can be associated with a test: * time-out delay (used by test driver to detect infinite loop and dead-lock) * description .. _stringize: Flexible conversion of object to string for assertion ===================================================== The framework provides a specific service to convert an object into a string. This service is used by assertions to generate diagnostic. Two operations are provided by the service: * conversion of a string into the standard std::string type. * conversion of an object into a string. If the object is a string type it is automatically quoted and escaped. While the default implementation relies on std::ostream for conversion of object to string, an alternative implementation can easily be provided (for platform such eVC++ 4 which do not provide iostream). If the target compiler supports function template partial specialization and the feature is enabled in the configuration, pointer will be automatically deferenced. Flexible test for equality for assertion ======================================== The framework provides a specific service to check if two objects are equal. If the target compiler supports function template partial specialization and the feature is enabled in the configuration, pointer will be automatically deferenced. [Notes: current implementation should be enhanced to detect if a specific overload is provided for comparison of AType and BType is provided, and automatically fallback on the generic == otherwise. This is similar to the convert to string trick used by stringize().] .. _`enumeration and string assertions`: Advanced assertions providing detailed diagnostic ================================================= Two advanced assertions facility are provided: * enumeration based assertions * string based assertions Enumeration based assertions ---------------------------- The enumeration based assertions works with the concept of enumerator. An enumerator implements a specific interface allowing enumeration of the elements of a collection. The CppTL library provides a complete implementation of enumerator that supports STL containers and iterators and can be very easily extended to support other container interfaces. The following assertions are provided for enumeration: * ensure that two enumerations have the same elements in the same order (sequence). On failure, the position of the first different element is reported, as well as the list of identical elements, and the remaining part of the expected and actual lists of elements. * ensure that two enumerations have the same elements regardless of the order (set). On failure, the list of extraneous and missing elements is reported, as well as the list of actual and expected elements. *insert example here* String based assertions ----------------------- The following assertions are provided for string: * ensure two strings are equals. * ensure a string starts with an expected string. * ensure a string ends with an expected string. * ensure a string contains another expected string. String assertions works with heterogeneous string type and use the stringize_ mechanism provided by the framework to convert object to string. String assertions provide detailed diagnostics on failure, escaping the string over multiple lines when it contains \\n or \\r character. *insert example here* Create test cases using generic functor ======================================= Test cases delegate the actual task of setting up the test, running the test and cleaning up after the test (setUp/run/tearDown) to generic functors. A generic functor can be created from a C function, an object instance and a member function or from a functor. When a function with parameters used for testing, its parameters can be bound to specific value, hence creating a generic functor that requires no parameter. *insert example here* Resource Acquisition Is Initialization idiom (RAII) =================================================== The framework uses the fundamental *RAII* C++ idiom (Resource Acquisition Is Initialization idiom) for clarity. Smart-pointer are used to pass dynamically allocated pointer around. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: --- NEW FILE: index.txt --- ========= CppUnit 2 ========= :Author: Baptiste Lepilleur :Contact: bl...@us... .. _`Open Test`: opentest.html .. contents:: Document of interest for users ============================== * `CppUnit 2 features presentation <features.html>`__ * `Open Test presentation <opentest.html>`__ Document of interest for developpers ==================================== * `CppUnit 2 todo <cppunit2todo.html>`__ * `Open Test todo <opentesttodo.html>`__ .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: --- NEW FILE: default.css --- /* cascading style sheet for the HTML output of Docutils. */ .first { margin-top: 0 } .last { margin-bottom: 0 } a.toc-backref { text-decoration: none ; color: black } blockquote.epigraph { margin: 2em 5em ; } dd { margin-bottom: 0.5em } /* Uncomment (& remove this text!) to get bold-faced definition list terms dt { font-weight: bold } */ div.abstract { margin: 2em 5em } div.abstract p.topic-title { font-weight: bold ; text-align: center } div.attention, div.caution, div.danger, div.error, div.hint, div.important, div.note, div.tip, div.warning, div.admonition { margin: 2em ; border: medium outset ; padding: 1em } div.attention p.admonition-title, div.caution p.admonition-title, div.danger p.admonition-title, div.error p.admonition-title, div.warning p.admonition-title { color: red ; font-weight: bold ; font-family: sans-serif } div.hint p.admonition-title, div.important p.admonition-title, div.note p.admonition-title, div.tip p.admonition-title, div.admonition p.admonition-title { font-weight: bold ; font-family: sans-serif } div.dedication { margin: 2em 5em ; text-align: center ; font-style: italic } div.dedication p.topic-title { font-weight: bold ; font-style: normal } div.figure { margin-left: 2em } div.footer, div.header { font-size: smaller } div.sidebar { margin-left: 1em ; border: medium outset ; padding: 0em 1em ; background-color: #ffffee ; width: 40% ; float: right ; clear: right } div.sidebar p.rubric { font-family: sans-serif ; font-size: medium } div.system-messages { margin: 5em } div.system-messages h1 { color: red } div.system-message { border: medium outset ; padding: 1em } div.system-message p.system-message-title { color: red ; font-weight: bold } div.topic { margin: 2em } h1.title { text-align: center } h2.subtitle { text-align: center } hr { width: 75% } ol.simple, ul.simple { margin-bottom: 1em } ol.arabic { list-style: decimal } ol.loweralpha { list-style: lower-alpha } ol.upperalpha { list-style: upper-alpha } ol.lowerroman { list-style: lower-roman } ol.upperroman { list-style: upper-roman } p.attribution { text-align: right ; margin-left: 50% } p.caption { font-style: italic } p.credits { font-style: italic ; font-size: smaller } p.label { white-space: nowrap } p.rubric { font-weight: bold ; font-size: larger ; color: maroon ; text-align: center } p.sidebar-title { font-family: sans-serif ; font-weight: bold ; font-size: larger } p.sidebar-subtitle { font-family: sans-serif ; font-weight: bold } p.topic-title { font-weight: bold } pre.address { margin-bottom: 0 ; margin-top: 0 ; font-family: serif ; font-size: 100% } pre.line-block { font-family: serif ; font-size: 100% } pre.literal-block, pre.doctest-block { margin-left: 2em ; margin-right: 2em ; background-color: #eeeeee } span.classifier { font-family: sans-serif ; font-style: oblique } span.classifier-delimiter { font-family: sans-serif ; font-weight: bold } span.interpreted { font-family: sans-serif } span.option { white-space: nowrap } span.option-argument { font-style: italic } span.pre { white-space: pre } span.problematic { color: red } table { margin-top: 0.5em ; margin-bottom: 0.5em } table.citation { border-left: solid thin gray ; padding-left: 0.5ex } table.docinfo { margin: 2em 4em } table.footnote { border-left: solid thin black ; padding-left: 0.5ex } td, th { padding-left: 0.5em ; padding-right: 0.5em ; vertical-align: top } th.docinfo-name, th.field-name { font-weight: bold ; text-align: left ; white-space: nowrap } h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt { font-size: 100% } tt { background-color: #eeeeee } ul.auto-toc { list-style-type: none } --- NEW FILE: builddoc.bat --- rst2html cppunit2todo.txt cppunit2todo.html rst2html index.txt index.html rst2html opentest.txt opentest.html rst2html opentesttodo.txt opentesttodo.html rst2html features.txt features.html --- NEW FILE: cppunit2todo.txt --- ============== CppUnit 2 todo ============== :Author: Baptiste Lepilleur :Contact: bl...@us... .. contents:: .. _TestDriver: opentest.html .. _TestRunner: opentest.html .. _`Open Test`: opentest.html Moving CppUnit 2 to Open Test ============================= Starting point -------------- Currently, the framework has adopted CppUnit approach to driving testing. This means that the framework provides services for both TestDriver_ and TestRunner_ responsibility. Since CppUnit is a testing framework, it should only provide the TestRunner_ for integration with `Open Test`_. What should be removed? ----------------------- * All stuffs concerning test reporting, output result must be removed (will likely be moved to OpenTest but will no longer be based on CppUnit classes). Should at least include the following headers: * progresslistener.h * properties.h (should use OpenTest properties or some proxy) * testcontext.h (some stuff need to be preserved, see `What need to be changed?`_) * testlistener.h * testresult.h (preserved some stuff, integrate some to the TestRunner) * testrunresult.h What need to be changed? ------------------------ * Test should no longer be runnable * TestContext guard chain and protect functions should be moved to the TestRunner. The rest should be taken out => no more TestListener. * TestResult should be integrated to TestRunner. Features to add =============== Test descriptive information ---------------------------- Provides a way to associate descriptive information to a test. **Warning**: it should be designed/documented so that only the framework uses it to declare test. Should also find a way to integrate this with test fixture declaration. Test configuration information ------------------------------ Just before test execution, test descriptive information should be merged with test configuration information (configuration information overriding test descriptive information if they share properties). A mean should be provided to the user to access a read only copy of those data. Test result information ----------------------- The user should be able to set test result information during the test run. This can be used to add a line of log, set specific message, actual values, assertion location... Logging stream -------------- Provides a logging stream that associated logged data to the test. Logged data should be sent to the TestDriver on each '\\n'. This probably requires implementing a streambuffer. This logging stream would be a singleton, like std::cout => CppUT::log. Data written in the stream should be forwarded to the component managing TestInfo (TestRunner ?). This ensures that the correct threading strategy is used (one log per thread, one log for all thread...). Checking assertions vs. aborting assertions ------------------------------------------- Checking assertions (opposed to aborting assertions) need to be added. This should use the same mechanism as `setCheckPointLocation()`: * `setAssertionType( checkingAssertion )` * `setAssertionType( abortingAssertion )` The `fail` function checks the assertion type to determine if an assertion if thrown of is an assertion occurred. It might also be interesting to see if removing Message is worth it. It could be replaced by direct update of the current assertion `Properties`. Test plug-in wrapper application -------------------------------- A TestRunner wrapper application needs to be written for test plug-in support with Open test. The wrapper application: * load the DLL * call a specific function on the DLL passing remaining command-line parameter .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: |