File	Date	Author	Commit
testset	2017-09-12	David Tardon	[d567dd] drop executable bit
.gitignore	2010-07-22	William Lachance	[8b6009] .cvsignore -> .gitignore
README	2004-06-10	Marc Maurer	[1b003e] Updated readme with the new callgraph test
regenerate_raw.pl	2013-11-07	Fridrich Štrba	[4d8ce0] Canonicalize xml so that the regression be poss...
regression.pl	2013-11-07	Fridrich Štrba	[4d8ce0] Canonicalize xml so that the regression be poss...

Read Me

About Regression Testing

A fully automated regression test for libwpd can be done by calling

	./regression.pl
	
The results of the test are displayed on STDOUT, and can be in plain text format
or HTML, depending on the value of the $html variable located near the top
of the regression.pl script.

* Test performed
Two types of tests are performed for each WordPerfect document that is in the 
test set : a 'raw diff' test and a 'call graph' test. The tests performed by
each test type are explained below.

* Adding a new document to the test set
To add a document to the test set, simply copy the file into the 
./testset/WP4, ./testset/WP5 or ./testset/WP6 directory, depending on the 
document type. Update the 'regression.in' file (each of these directories has
its own copy) with an entry for the newly added file. Finally, store the 
output of wpd2raw for this new document into a file called 
'<new wordperfect file>.raw'. Be sure to use the latest version of wpd2raw.


Raw Diff Test
=============

The raw diff test is a simple test to see if the output of wpd2raw for a certain
file has changed, for example after changes made to libwpd.

This test will compare the current output of wpd2raw against a stored version of 
the output. The comparison is done using the 'diff' command.

Note that whenever the raw diff test reports a change, it does not necessarily
mean that an error has been found. Tree cases of a change can be identified:

1) libwpd has been changed, for example to fix a bug or to add a new feature.
A change reported by the raw diff test should be examined closely to determine
if the change includes ONLY the expected result of the change to libwpd. If it 
includes some other unexpected/unwanted changes in the output as well, a 
regression has been found.

2) the output of wpd2raw has been changed, for example to add a new feature.
The raw diff test will now report a change for all files using this feature.
Again, no real harm has been done, and thus the change should not be considered
as an error.

Whenever the output of wpd2raw changes as expected (due to changes to libwpd or 
wpd2raw itself), the raw output for every file in the test set should be 
regenerated and stored. This process is automated, and can be done simply by
calling:

	./regenerate_raw.pl
	
After the raw output has been updated, the changes should be stored in cvs.


Call Graph Test
===============

A call graph test checks if every 'open' callback is closed by a 'close' 
callback. Eg. Suppose wpd2raw would generate the following output for a 
certain file:

openDocument()
  openPageSpan()
    openSection()
      openParagraph()
        insertText()
      closeParagraph()
    closeSection()
  closePageSpan()
closeDocument()

Nothing is wrong here: every 'open' callback is followed properly with a 
corresponding 'close' callback. Now, suppose the wpd2raw would generate the 
following output instead:

openDocument()
  openPageSpan()
    openSection()
      openParagraph()
        insertText()
      closeParagraph()
    closeSection()
  closeDocument()

We clearly see that the openPageSpan() callback is never followed by a 
closePageSpan() callback, which is wrong.

The Call Graph Test will report the number of unmatched close/open calls. It 
will be clear that a result of '0' indicates 'no error'. A number less than 
'0' means more 'close' callbacks are called than the number of 'open' callbacks.
A result less greater than '0' means more 'open' callbacks are called than the
number of 'close' callbacks.

Finally, suppose the wpd2raw would generate the following output:

openDocument()
  openPageSpan()
    openSection()
      openParagraph()
        insertText()
      closeTable()
    closePageSpan()
  closeSection()
closeDocument()

An inspection of this output reveals that every 'open' callback is followed
by a matching 'close' callback. However, the sequence in which these callbacks
are closed is wrong: the closePageSpan() callback is executed before the 
closeSection() callback!


Valgrind  Test
==============

TODO: write me