To improve build time testing (e.g. done in the (nearly finished) Debian package) it would be nice if the following features would be implemented:
top level Makefile aware of (some of) the tests/Makefile targets.
shipping of the reference files in the pasdoc package.
ensure non-zero return value if results are not as expected. Currently the tests pass, even when pasdoc, xmllint and onsgmls are not on the path or installed.
Agreed with all, especially the 2nd point ("reference files in the pasdoc package"). Long time ago, when I implemented these tests, I did not want to place reference files in SVN (as I didn't want to place so many auto-generated files in SVN or tar.gz sources), that is why tests are downloaded by special script scripts/download_correct_tests_output.sh . Now I think that just placing the reference files in SVN would make the process easier (you only download SVN).
And integrating this with main Makefile would encourage to update the reference files always, and run checks more often.
Compressed with gz, the reference files should not weight > 1 MB, looking at current sizes. So size is not a problem.
I'll tackle this when I have time. Or, patches/commits are welcome :)
All done:)
AD 2 ("shipping of the reference files in the pasdoc package") - The correct tests output is now trivially kept inside our code repository (GIT on https://github.com/pasdoc/pasdoc , auto-synchronized with SVN repository on SourceForge). In
tests/testcases_output/
, documented in tests/README.md, see https://github.com/pasdoc/pasdoc/tree/master/tests .It was more critical now, as SourceForge doesn't allow creating shells anymore, which was breaking our "upload_correct_tests_output.sh" script.
The new solution avoids any downloading/uploading on SourceForge servers. And no packing to tar.gz. We can also easily check changes by just
git diff
, and commit them withgit commit
. So, overall, we avoid maintaining some of our scripts, and hopefully the simplicity of the new solution is a win :)AD 1 ("top level Makefile aware of (some of) the tests/Makefile targets.") - To run all the tests, just execute
make tests
now. This is a trivial shortcut to executingrun_all_tests.sh
inside thetests/
subdirectory.Note that it compares correctness using
git diff --exit-code testcases_output/
by default. If you have sources without the.git
repository data, then define USE_DIFF_TO_COMPARE=true first, like:AD 3 ("ensure non-zero return value if results are not as expected") - I believe this was solved by your recent patch that added
which
checks to the scripts :)