File	Date	Author	Commit
src	2013-11-17	David Tardon	[4b1012] add test case sources too
testset	2018-01-01	David Tardon	[659d77] regenerate
README	2013-11-16	David Tardon	[e3cb94] create regression test suite
regenerate_raw.pl	2015-12-24	David Tardon	[2c34ae] enable tests for more Palm formats
regression.pl	2015-12-24	David Tardon	[2c34ae] enable tests for more Palm formats

Read Me

About Regression Testing

A fully automated regression test for libe-book can be done by calling

	./regression.pl

The results of the test are displayed on STDOUT, and can be in plain text format
or HTML, depending on the value of the $html variable located near the top
of the regression.pl script.

* Test performed
Two types of tests are performed for each WordPerfect document that is in the
test set : a 'raw diff' test and a 'call graph' test. The tests performed by
each test type are explained below.

* Adding a new document to the test set
To add a document to the test set, simply copy the file into the
./testset/WP4, ./testset/WP5 or ./testset/WP6 directory, depending on the
document type. Update the 'regression.in' file (each of these directories has
its own copy) with an entry for the newly added file. Finally, store the
output of ebook2raw for this new document into a file called
'<new wordperfect file>.raw'. Be sure to use the latest version of ebook2raw.


Raw Diff Test
=============

The raw diff test is a simple test to see if the output of ebook2raw for a certain
file has changed, for example after changes made to libe-book.

This test will compare the current output of ebook2raw against a stored version of
the output. The comparison is done using the 'diff' command.

Note that whenever the raw diff test reports a change, it does not necessarily
mean that an error has been found. Tree cases of a change can be identified:

1) libe-book has been changed, for example to fix a bug or to add a new feature.
A change reported by the raw diff test should be examined closely to determine
if the change includes ONLY the expected result of the change to libe-book. If it
includes some other unexpected/unwanted changes in the output as well, a
regression has been found.

2) the output of ebook2raw has been changed, for example to add a new feature.
The raw diff test will now report a change for all files using this feature.
Again, no real harm has been done, and thus the change should not be considered
as an error.

Whenever the output of ebook2raw changes as expected (due to changes to libe-book or
ebook2raw itself), the raw output for every file in the test set should be
regenerated and stored. This process is automated, and can be done simply by
calling:

	./regenerate_raw.pl

After the raw output has been updated, the changes should be stored in cvs.


Call Graph Test
===============

A call graph test checks if every 'open' callback is closed by a 'close'
callback. Eg. Suppose ebook2raw would generate the following output for a
certain file:

openDocument()
  openPageSpan()
    openSection()
      openParagraph()
        insertText()
      closeParagraph()
    closeSection()
  closePageSpan()
closeDocument()

Nothing is wrong here: every 'open' callback is followed properly with a
corresponding 'close' callback. Now, suppose the ebook2raw would generate the
following output instead:

openDocument()
  openPageSpan()
    openSection()
      openParagraph()
        insertText()
      closeParagraph()
    closeSection()
  closeDocument()

We clearly see that the openPageSpan() callback is never followed by a
closePageSpan() callback, which is wrong.

The Call Graph Test will report the number of unmatched close/open calls. It
will be clear that a result of '0' indicates 'no error'. A number less than
'0' means more 'close' callbacks are called than the number of 'open' callbacks.
A result less greater than '0' means more 'open' callbacks are called than the
number of 'close' callbacks.

Finally, suppose the ebook2raw would generate the following output:

openDocument()
  openPageSpan()
    openSection()
      openParagraph()
        insertText()
      closeTable()
    closePageSpan()
  closeSection()
closeDocument()

An inspection of this output reveals that every 'open' callback is followed
by a matching 'close' callback. However, the sequence in which these callbacks
are closed is wrong: the closePageSpan() callback is executed before the
closeSection() callback!


Valgrind  Test
==============

TODO: write me