[Cppunit-cvs] cppunit2/doc builddoc.bat,NONE,1.1 cppunit2todo.txt,NONE,1.1 default.css,NONE,1.1 features.txt,NONE,1.1 index.txt,NONE,1.1 opentest.txt,NONE,1.1 opentesttodo.txt,NONE,1.1

[Baptiste L. <bl...@us...>]

Update of /cvsroot/cppunit/cppunit2/doc
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv22756

Added Files:
	builddoc.bat cppunit2todo.txt default.css features.txt 
	index.txt opentest.txt opentesttodo.txt 
Log Message:
* added some documentations: features, open test, todo for cppunit 2 and open test. docutils is needed to build the doc.

--- NEW FILE: opentest.txt ---
======================
Open Test presentation
======================

:Author: Baptiste Lepilleur
:Contact: bl...@us...

.. contents::

What is Open Test?
==================

Open Test provides a mean to easily integrate a testing framework
with Open Test `test drivers`__.

__ TestDriver_

The open test concept emerged from the complete separation of the
way tests of running test from driving and monitoring the test run
in CppUnit 2.

Those two activities are managed by two components: the TestRunner_ and 
the TestDriver_.

While implementing a TestRunner_ is cheap, implementing TestDriver_ cost a lot, 
especially if the test driver must provide a graphical user interface.

With Open Test, it is possible to reuse the existing TestDriver_ and 
implement a TestRunner_ for a specific test framework. This makes 
implementing a specific functional test framework cheap since you only need to
implement the testing part. All the monitoring, user interface parts are
already implemented and reusable.

.. _TestRunner:

What is a TestRunner?
=====================

A test runner is a lightweight component that provides the following
services:

* declare all the available tests
* run a specific list of tests and track the run of each test.

[Notes: need to be enhanced, resource acquisition service...]


.. _TestDriver:

What is a TestDriver?
=====================

A test driver as many responsibility:

* monitoring the test run and detecting crash and infinite-loop to resume 
  the test run.
* selecting the test(s) to run
* providing feedback during the test run (what test is being run, did the test
  pass...)
* generating report of the test run (output result in XML, compiler and human
  friendly text format...)

A test driver can usually be configured from both the command-line and a
configuration file for automated testing.

Graphical test drivers are provided for interactive browsing of the
test hierarchy, and visual feedback...

It should be noted that some test drivers could be connected to multiple
test runner at the same time.

How does the TestDriver communicate with the TestRunner?
========================================================

While it is possible for a TestDriver_ to directly call methods of a
TestRunner_, it is not the way it is usually done. Implementing reliable
test run monitoring requires running the test in a separate process.

Instead of going the easy way and using shared memory to share test data
structure, communication between the TestDriver_ and a TestRunner_ rely on
a duplex stream communication. While a little more complex to implement,
this have the great advantage that a TestDriver_ can be connected to
any TestRunner_ that implements the same protocol.

A TestDriver_ can communicate with a TestRunner_ that:

* is not using the same run-time version of the C library (a common issue
  on the Windows platform)
* is not compiled using the same compiler
* is running on a different architecture (is a duplex network connection
  is used)
* is possibly implemented using another language (would require porting
  the communication layer to the alternate language)


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:


--- NEW FILE: opentesttodo.txt ---
==============
Open Test todo
==============

:Author: Baptiste Lepilleur
:Contact: bl...@us...

.. contents::

.. _TestDriver: opentest.html
.. _TestRunner: opentest.html
.. _`Open Test`: opentest.html

General status
==============

`Open Test`_ is a fairly new concept and hence has a low level of maturity.
Everything is to be done.

The TestRunner_ interface has been defined and implemented for CppUnit 2.

Everything related to test driver need to be defined and implemented. This
involves:

* defining property tree for suite and test declaration, test runner 
  configuration and test result.
* writing a simple text based TestDriver_
* test monitoring and duplex communication through stream
* test driver xml output
* extend the text based TestDriver_ to accept multiple TestRunner
* GUI based TestDriver_
* allow TestDriver selection from command-line

Property tree structure
=======================

Suite and test declarations
---------------------------

TestDeclarator::beginSuite() and TestDeclarator::addTest().

* time-out delay
* description,
* input/expected values (for functional testing based on input/output)

Test runner configuration
-------------------------

TestRunner::runTests().

* is logging enabled
* number of threads

Test configuration
------------------

TestPlanEntry::configuration().

Two categories of data are associated to a test at this stage of the testing:

* test specific runner configuration
* test descriptive data

Test specific runner configuration:

* enable logging 

Test descriptive data:

* input/expected values override

Test execution result
---------------------

TestRunTracker::addTestInfo().

Notes: this method probably needs to be modified since it can be called
multiple times (for example, every time a line of log is added).

* test status: success ?
* test status type: success/failure/fault...
* test result message
* test result source code location
* test result (input/) actual/expected
* test result log

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:

Simple text based TestDriver
============================

Implementing a simple text based TestDriver is required to validate the 
interface of the TestRunner. It should use direct method call for 
communication with the TestRunner (which means no monitoring).

Features:

* text progress (like CppUnit BriefProgressListener)
* dynamic log output as the test runner
* compiler output (assertion location in a compiler compatible format with 
  failure detail below).
* optionally select test to run from the command-line

Status: partially implemented in opentest/texttestdriver.h.

Test monitoring
===============

The test monitoring feature needs to be implemented. It requires that
the TestDriver_ spawn a child process that run the TestRunner_ and
communicate with it.

Communication between the TestDriver_ and the TestRunner_ should be stream
based as it allows the usage of various implementation, such as 
shared memory, pipe, and network... Ideally, all implementations should provide
the same interface, only requiring the TestDriver to bother about the
implementation type at connection time.

A spike solution is being worked on to explore shared memory on Windows 
(Baptiste: a working implementation is in place but need a lot of clean-up).

GUI based TestDriver
====================

* should provide all the functionalities of a text based TestDriver
  (it simply does not close the result window when the test are done).
* should support for multiple TestRunner
* allow loading of multiple test plug-in (spawn test plug-in wrapper 
  application for each DLL)

QT will likely be the first GUI toolkit to be ported as it is multi-platform
and is easier to implement than MFC or wxWidget.

TestDriver selection from command-line
======================================

It should be possible to select the TestDriver to use from the command-line.
This implies that all test drivers provide a similar interface.

The user should be able to choose between text driver and GUI driver from
the command-line (use plug-in for TestDriver ?).

--- NEW FILE: features.txt ---
===============================
CppUnit 2 features presentation
===============================

:Author: Baptiste Lepilleur
:Contact: bl...@us...

.. _`Open Test`: opentest.html

.. contents::

Supports for quick develop/test cycle
=====================================

The framework help having a quick develop/test cycle with the following 
features:

* test factory registry to reduce build time (only the modified test 
  source file need to be recompiled)
* compiler format source location of failed assertion to jump straight
  to the assertion in the source code
* detailed diagnostic on assertion failure, frequently avoiding the
  need to run the debugger
* test plug-in to reduce link-time on large project

In a well-tuned environment that cycle can be as short as a few seconds,
as you usually only recompile only two translation units (the test and
the tested implementation), and run the test automatically after a 
successful build. This makes test driven design possible in C++.

Component based testing with test plug-in
=========================================

Tests can be embedded into the dynamic library to test. An application
provided with the framework is used to run the test contained in a
dynamic library. The application can run the test of a
specific dynamic library or of multiple dynamic libraries at once.

Test-plug in also help separating the tests from how they are run, 
thus allowing to centralize the way tests are run and their result
reported.

Test monitor to detect crash and infinite loop
==============================================

The framework provides easy mechanism to monitor the test run. It can 
automatically spawn a child process that run the test and monitor it
for crash or infinite loop. In the case of infinite loop it terminates
the child process. If either a crash or an infinite loop occurred, a
new child process is spawned and testing resume after the problematic
test case.

Graphical test driver
=====================

Graphical test driver are provided to:

* navigate test hierarchy
* run a specific test or test suite
* browse test result

Open test runner interface
==========================

The test drivers (the components that run and monitor the tests) can work
with any test runner that implements the `Open Test`_ interface. This
means that the framework graphical test driver can be used to run and 
monitor test implemented using another test framework as long as 
the `Open Test`_ interface can be implemented for that other framework.

More detail can be found in the `Open Test`_ document.

Flexible output of test result
==============================

The result of a test run is thoroughly inspectable. Outputter are 
provided to output the result in multiple format, such as XML for
integration in a tool chain, or a human readable format with
compiler like assertion location for integration with IDE.

Extensive test driver configuration
===================================

Test drivers can be configured either using the command-line or
persistent configuration files. 

The configuration management is designed in such a way that new
configuration options can easily be added.

Custom assertion support
========================

The assertion design separate the assertion location tracking
from the function call used to implement the assertion.

This means that assertion functions can have a variable number of 
parameters, and that default parameters are allowed.

Here is an example of how the assertion ``CPPUT_FAIL( "bad parameters." )`` 
is expanded.

The assertion is defined as follow: ::

	# define CPPUT_FAIL              \
	   CPPUT_BEGIN_ASSERTION_MACRO() \
	   ::CppUT::fail

Which is expanded into: ::

	::CppUT::setCheckPointLocation( 
		::CppUT::Location( __FILE__, __LINE__ ) ), // notes the comma
	::CppUT::fail

Hence, the code ``CPPUT_FAIL( "bad parameters." )`` first call 
``SetCheckPointLocation`` to memorize where the assertion occurred in
the source code, and then call the function ``fail`` with the parameter(s)
specified by the user. As the parameters are not part of the macro,
their number can vary.

One of the advantages of writing custom assertions is that in addition
to detecting that the assertion failed, a detailed diagnostic can be
provided. For example, when comparing two structured trees, 
knowing that they are different let you know that something is wrong, 
but not *what*. With custom assertions, the assertion can be tested, and
a detailed diagnostic based on the knowledge of what the structure
represent and what would be useful to diagnostic the failure can
be reported.

An application of advanced assertions can be found in 
`enumeration and string assertions`_.

Non aborting assertions support
===============================

The framework provides both assertions that abort the current test (throwing
an exception), and assertions that do not abort the current test but 
keep track of the failures. Those assertions are named respectively 
"aborting assertions " and "checking assertions".

The implementation of this feature relies on a mechanism similar to 
`setCheckPointLocation()` described in `Custom assertion support`_, 
meaning that all aborting assertions can easily be made available as 
checking assertions and vice versa.

Property tree as test result for integration
============================================

The result of each test case run is always a property tree. All data
related to the execution of a test case are stored as property in this
tree. For instance, the status of the test (success, failure...)
is just a specific property in this tree.

Using a property tree provide a lot of flexibility as it allows
association of complex data to each test run such as:

* duration of the test
* log messages
* alternative test execution status (succeeded with warning, failure
  because of missing resources...)
* actual and expected value

Being able to report actual and expected values in a test result is a
very powerful features. For example, if you are generating input values
from a spreadsheet application, after running the test, you can put the
actual values back into the spreadsheet.

Property tree associated to each test declaration
=================================================

Each test has a property tree. The property tree can be used to store
descriptive data for each test. Property can be marked as "inheritable",
meaning all tests of a test suite inherit this property (but they can
override it).

Here is some example of properties that can be associated with a test:

* time-out delay (used by test driver to detect infinite loop and dead-lock)
* description

.. _stringize:

Flexible conversion of object to string for assertion
=====================================================

The framework provides a specific service to convert an object into a 
string. This service is used by assertions to generate diagnostic.

Two operations are provided by the service:

* conversion of a string into the standard std::string type.
* conversion of an object into a string. If the object is a string 
  type it is automatically quoted and escaped.

While the default implementation relies on std::ostream for conversion
of object to string, an alternative implementation can easily be 
provided (for platform such eVC++ 4 which do not provide iostream).

If the target compiler supports function template partial specialization
and the feature is enabled in the configuration, pointer will be
automatically deferenced.

Flexible test for equality for assertion
========================================

The framework provides a specific service to check if two objects are
equal.

If the target compiler supports function template partial specialization
and the feature is enabled in the configuration, pointer will be
automatically deferenced.

[Notes: current implementation should be enhanced to detect if a 
specific overload is provided for comparison of AType and BType
is provided, and automatically fallback on the generic == otherwise.
This is similar to the convert to string trick used by stringize().]

.. _`enumeration and string assertions`:

Advanced assertions providing detailed diagnostic
=================================================

Two advanced assertions facility are provided:

* enumeration based assertions
* string based assertions

Enumeration based assertions
----------------------------

The enumeration based assertions works with the concept of enumerator.
An enumerator implements a specific interface allowing enumeration
of the elements of a collection. The CppTL library provides a complete
implementation of enumerator that supports STL containers and iterators
and can be very easily extended to support other container interfaces.

The following assertions are provided for enumeration:

* ensure that two enumerations have the same elements in the same order 
  (sequence). On failure, the position of the first different element
  is reported, as well as the list of identical elements, 
  and the remaining part of the expected and actual lists of elements.
* ensure that two enumerations have the same elements regardless 
  of the order (set). On failure, the list of extraneous and missing
  elements is reported, as well as the list of actual and expected 
  elements.

*insert example here*


String based assertions
-----------------------

The following assertions are provided for string:

* ensure two strings are equals.
* ensure a string starts with an expected string.
* ensure a string ends with an expected string.
* ensure a string contains another expected string.

String assertions works with heterogeneous string type and use the
stringize_ mechanism provided by the framework to convert object to string.

String assertions provide detailed diagnostics on failure, escaping the
string over multiple lines when it contains \\n or \\r character.

*insert example here*

Create test cases using generic functor
=======================================

Test cases delegate the actual task of setting up the test, running the test and 
cleaning up after the test (setUp/run/tearDown) to generic functors.

A generic functor can be created from a C function, an object instance and 
a member function or from a functor. When a function with parameters
used for testing, its parameters can be bound to specific value, hence
creating a generic functor that requires no parameter.

*insert example here*

Resource Acquisition Is Initialization idiom (RAII)
===================================================

The framework uses the fundamental *RAII* C++ idiom 
(Resource Acquisition Is Initialization idiom) for clarity.

Smart-pointer are used to pass dynamically allocated pointer around.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:


--- NEW FILE: index.txt ---
=========
CppUnit 2
=========

:Author: Baptiste Lepilleur
:Contact: bl...@us...

.. _`Open Test`: opentest.html

.. contents::

Document of interest for users
==============================

* `CppUnit 2 features presentation <features.html>`__
* `Open Test presentation <opentest.html>`__

Document of interest for developpers
====================================

* `CppUnit 2 todo <cppunit2todo.html>`__
* `Open Test todo <opentesttodo.html>`__

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:


--- NEW FILE: default.css ---
/*
cascading style sheet for the HTML output of Docutils.
*/

.first {
  margin-top: 0 }

.last {
  margin-bottom: 0 }

a.toc-backref {
  text-decoration: none ;
  color: black }

blockquote.epigraph {
  margin: 2em 5em ; }

dd {
  margin-bottom: 0.5em }

/* Uncomment (& remove this text!) to get bold-faced definition list terms
dt {
  font-weight: bold }
*/

div.abstract {
  margin: 2em 5em }

div.abstract p.topic-title {
  font-weight: bold ;
  text-align: center }

div.attention, div.caution, div.danger, div.error, div.hint,
div.important, div.note, div.tip, div.warning, div.admonition {
  margin: 2em ;
  border: medium outset ;
  padding: 1em }

div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title {
  color: red ;
  font-weight: bold ;
  font-family: sans-serif }

div.hint p.admonition-title, div.important p.admonition-title,
div.note p.admonition-title, div.tip p.admonition-title,
div.admonition p.admonition-title {
  font-weight: bold ;
  font-family: sans-serif }

div.dedication {
  margin: 2em 5em ;
  text-align: center ;
  font-style: italic }

div.dedication p.topic-title {
  font-weight: bold ;
  font-style: normal }

div.figure {
  margin-left: 2em }

div.footer, div.header {
  font-size: smaller }

div.sidebar {
  margin-left: 1em ;
  border: medium outset ;
  padding: 0em 1em ;
  background-color: #ffffee ;
  width: 40% ;
  float: right ;
  clear: right }

div.sidebar p.rubric {
  font-family: sans-serif ;
  font-size: medium }

div.system-messages {
  margin: 5em }

div.system-messages h1 {
  color: red }

div.system-message {
  border: medium outset ;
  padding: 1em }

div.system-message p.system-message-title {
  color: red ;
  font-weight: bold }

div.topic {
  margin: 2em }

h1.title {
  text-align: center }

h2.subtitle {
  text-align: center }

hr {
  width: 75% }

ol.simple, ul.simple {
  margin-bottom: 1em }

ol.arabic {
  list-style: decimal }

ol.loweralpha {
  list-style: lower-alpha }

ol.upperalpha {
  list-style: upper-alpha }

ol.lowerroman {
  list-style: lower-roman }

ol.upperroman {
  list-style: upper-roman }

p.attribution {
  text-align: right ;
  margin-left: 50% }

p.caption {
  font-style: italic }

p.credits {
  font-style: italic ;
  font-size: smaller }

p.label {
  white-space: nowrap }

p.rubric {
  font-weight: bold ;
  font-size: larger ;
  color: maroon ;
  text-align: center }

p.sidebar-title {
  font-family: sans-serif ;
  font-weight: bold ;
  font-size: larger }

p.sidebar-subtitle {
  font-family: sans-serif ;
  font-weight: bold }

p.topic-title {
  font-weight: bold }

pre.address {
  margin-bottom: 0 ;
  margin-top: 0 ;
  font-family: serif ;
  font-size: 100% }

pre.line-block {
  font-family: serif ;
  font-size: 100% }

pre.literal-block, pre.doctest-block {
  margin-left: 2em ;
  margin-right: 2em ;
  background-color: #eeeeee }

span.classifier {
  font-family: sans-serif ;
  font-style: oblique }

span.classifier-delimiter {
  font-family: sans-serif ;
  font-weight: bold }

span.interpreted {
  font-family: sans-serif }

span.option {
  white-space: nowrap }

span.option-argument {
  font-style: italic }

span.pre {
  white-space: pre }

span.problematic {
  color: red }

table {
  margin-top: 0.5em ;
  margin-bottom: 0.5em }

table.citation {
  border-left: solid thin gray ;
  padding-left: 0.5ex }

table.docinfo {
  margin: 2em 4em }

table.footnote {
  border-left: solid thin black ;
  padding-left: 0.5ex }

td, th {
  padding-left: 0.5em ;
  padding-right: 0.5em ;
  vertical-align: top }

th.docinfo-name, th.field-name {
  font-weight: bold ;
  text-align: left ;
  white-space: nowrap }

h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt {
  font-size: 100% }

tt {
  background-color: #eeeeee }

ul.auto-toc {
  list-style-type: none }

--- NEW FILE: builddoc.bat ---
rst2html cppunit2todo.txt cppunit2todo.html
rst2html index.txt index.html
rst2html opentest.txt opentest.html
rst2html opentesttodo.txt opentesttodo.html
rst2html features.txt features.html


--- NEW FILE: cppunit2todo.txt ---
==============
CppUnit 2 todo
==============

:Author: Baptiste Lepilleur
:Contact: bl...@us...

.. contents::

.. _TestDriver: opentest.html
.. _TestRunner: opentest.html
.. _`Open Test`: opentest.html


Moving CppUnit 2 to Open Test
=============================

Starting point
--------------

Currently, the framework has adopted CppUnit approach to driving testing. 
This means that the framework provides services for both TestDriver_ and
TestRunner_ responsibility.

Since CppUnit is a testing framework, it should only provide the 
TestRunner_ for integration with `Open Test`_.

What should be removed?
-----------------------

* All stuffs concerning test reporting, output result must be removed
  (will likely be moved to OpenTest but will no longer be based on
  CppUnit classes).

Should at least include the following headers:

* progresslistener.h
* properties.h  (should use OpenTest properties or some proxy)
* testcontext.h (some stuff need to be preserved, see `What need to be changed?`_)
* testlistener.h
* testresult.h (preserved some stuff, integrate some to the TestRunner)
* testrunresult.h

What need to be changed?
------------------------

* Test should no longer be runnable
* TestContext guard chain and protect functions should be moved
  to the TestRunner. The rest should be taken out => no more TestListener.
* TestResult should be integrated to TestRunner.

Features to add
===============

Test descriptive information
----------------------------

Provides a way to associate descriptive information to a test.

**Warning**: it should be designed/documented so that only the framework 
uses it to declare test.

Should also find a way to integrate this with test fixture declaration.

Test configuration information
------------------------------

Just before test execution, test descriptive information should be merged
with test configuration information (configuration information overriding
test descriptive information if they share properties).

A mean should be provided to the user to access a read only copy of
those data.

Test result information
-----------------------

The user should be able to set test result information during the test
run.

This can be used to add a line of log, set specific message, 
actual values, assertion location...

Logging stream
--------------

Provides a logging stream that associated logged data to the test.

Logged data should be sent to the TestDriver on each '\\n'. This probably 
requires implementing a streambuffer.

This logging stream would be a singleton, like std::cout => CppUT::log. 

Data written in the stream should be forwarded to the component
managing TestInfo (TestRunner ?). This ensures that the correct threading
strategy is used (one log per thread, one log for all thread...).

Checking assertions vs. aborting assertions
-------------------------------------------

Checking assertions (opposed to aborting assertions) need to be added.

This should use the same mechanism as `setCheckPointLocation()`:

* `setAssertionType( checkingAssertion )`
* `setAssertionType( abortingAssertion )`

The `fail` function checks the assertion type to determine if an assertion
if thrown of is an assertion occurred.

It might also be interesting to see if removing Message is worth it. It could
be replaced by direct update of the current assertion `Properties`.

Test plug-in wrapper application
--------------------------------

A TestRunner wrapper application needs to be written for test plug-in
support with Open test.

The wrapper application:

* load the DLL
* call a specific function on the DLL passing remaining command-line parameter

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End: