#57 improve build time testing

[Paul Gevers]

To improve build time testing (e.g. done in the (nearly finished) Debian package) it would be nice if the following features would be implemented:

• top level Makefile aware of (some of) the tests/Makefile targets.

• shipping of the reference files in the pasdoc package.

• ensure non-zero return value if results are not as expected. Currently the tests pass, even when pasdoc, xmllint and onsgmls are not on the path or installed.

[Michalis Kamburelis]

Agreed with all, especially the 2nd point ("reference files in the pasdoc package"). Long time ago, when I implemented these tests, I did not want to place reference files in SVN (as I didn't want to place so many auto-generated files in SVN or tar.gz sources), that is why tests are downloaded by special script scripts/download_correct_tests_output.sh . Now I think that just placing the reference files in SVN would make the process easier (you only download SVN).

And integrating this with main Makefile would encourage to update the reference files always, and run checks more often.

Compressed with gz, the reference files should not weight > 1 MB, looking at current sizes. So size is not a problem.

I'll tackle this when I have time. Or, patches/commits are welcome :)

[Michalis Kamburelis]

All done:)

AD 2 ("shipping of the reference files in the pasdoc package") - The correct tests output is now trivially kept inside our code repository (GIT on https://github.com/pasdoc/pasdoc , auto-synchronized with SVN repository on SourceForge). In tests/testcases_output/, documented in tests/README.md, see https://github.com/pasdoc/pasdoc/tree/master/tests .

It was more critical now, as SourceForge doesn't allow creating shells anymore, which was breaking our "upload_correct_tests_output.sh" script.

The new solution avoids any downloading/uploading on SourceForge servers. And no packing to tar.gz. We can also easily check changes by just git diff, and commit them with git commit. So, overall, we avoid maintaining some of our scripts, and hopefully the simplicity of the new solution is a win :)

AD 1 ("top level Makefile aware of (some of) the tests/Makefile targets.") - To run all the tests, just execute make tests now. This is a trivial shortcut to executing run_all_tests.sh inside the tests/ subdirectory.

Note that it compares correctness using git diff --exit-code testcases_output/ by default. If you have sources without the .git repository data, then define USE_DIFF_TO_COMPARE=true first, like:

export USE_DIFF_TO_COMPARE=true
make tests

AD 3 ("ensure non-zero return value if results are not as expected") - I believe this was solved by your recent patch that added which checks to the scripts :)