A short status update on this thread:

1) There is a new wiki page, "Testing Gramps", which is currently a stub, indexing all my current knowledge about testing in Gramps.
Call for action: please review the page, and add anything relevant that you know, even if in outline form.
http://www.gramps-project.org/wiki/index.php?title=Testing_Gramps

2) The reports test runner script, runtest.sh, has been fixed and found 5 bugs, tagged with 'found-by-runtest.sh'. See the "attached issues" link from the tag page http://www.gramps-project.org/bugs/tag_view_page.php?tag_id=77

Two of the bugs have already been fixed by yours truly, and I have plans on fixing the remaining 3, but wait for various feedback there.

So this script does help gramps quality, and I strongly suggest everybody who performs changes in report-producing code, to play with it.

See more off the wiki page mentioned above.

3) RunAllTests.py, the unit test runner for unit test code located in test/ subdir, has also been fixed. 3 bugs I have fixed during that work relate to broken/outdated test code. One remaining bug, 6940, requires more work, and I'm waiting for input from Benny when he's back from the vacation, to understand what's going on there. Until it is fixed, this test runner script will show 1 failed test.

So if you feel like adding testing code to something you are developing/fixing now, you already can do it and run it using the "RunAllTests.py" runner.

4) Next time I've got some free cycles to work on this, I plan to revive the in-tree unit tests (e.g., the date_test.py, failing of which was mentioned earlier in this thread).

I believe that by getting the test toolset back in shape we'll be able to make gramps more stable.

VKh

On 25.06.2013 18:34, Vassilii Khachaturov wrote:
On 23.06.2013 17:02, Benny Malengier wrote:



2013/6/23 Vassilii Khachaturov <vassilii@tarunz.org>
Has anyone run any of the unit test scripts on trunk/gramps40?

vassilii@vassilii-VirtualBox:~/Gramps$ python
gramps/gen/lib/test/date_test.py
Traceback (most recent call last):
   File "gramps/gen/lib/test/date_test.py", line 48, in <module>
     from ...config import config
ValueError: Attempted relative import in non-package
vassilii@vassilii-VirtualBox:~/Gramps$ echo $PYTHONPATH
/home/vassilii/Gramps/gramps
vassilii@vassilii-VirtualBox:~/Gramps$ echo $GRAMPS_RESOURCES
/home/vassilii/Gramps

And the test suite runner scripts in test/ all refer to the gramps34/
era src/ directory...
Do I understand it correctly that none of the unit test scripts have
ever run on trunk/gramps40?

I'm about to develop some stuff that I'll need automatic tests to test,
so I wonder if anyone is working on reviving at least the test launcher
at the moment. Is there a bug tracker issue about it?

Ok, the test unit system was started, and probably worked for some time, but the peron/persons who wrote it wrongly assumed the other developers knew how to use it.
So, it would be really great if somebody revived it so that the work would not have been for nothing, but educating the other developers is a requirement if the system needs to actually work.
That means:

* documentation
* blog post on use, referral in the wiki
* a very, very, very simple script launcher to run the tests

Thanks for your answer!

I'll try to revive what I can on the technical level, and document on the go. First I'll start with gramps34 because, unfortunately, things got broken even there. Then when the existing test suites run again there, I'll carry that forward to the trunk, and also revive the additional test running code that exists there. I'll open the bugs as I go.

So far I have started with fixing the report testing suite which was blocked on non-existing reports. I have not even finished analyzing the results, but there are 2 small fixes committed already --- one to the test runner to fix it (i.e., bug in testing code), and another to a cli handler in the report cli plugin running code (bug production code). The impatient ones are welcome to see it in the bug tracker and the revision log. Further analysis might lead to filing more bugs. I'm using a special tag to indicate the bugs found by this test runner: found-by-run_tests.sh

I also believe that we shall need to at least add to the release procedures (and maybe to commit procedures) what tests must be run before something is released. Yes, I know there is an existing "test plan" document ;-)

For my own stuff in the past I used the method of adding a main section to the thing  being written. However, as that is not automated, it only ever remains maintained for the complex things like the dates, relationships, ...

*sigh* This thing actually started with me trying to run such a local non-automated test for date displayers, as you can see in the log above...

Vassilii