Thanks for leading this charge! Good stuff you've done already.
I am completely naive about testing, but a couple of comments:
1) Could the test files have the same names as what they are testing if
they are in a subdir? Such as: plugins/Calendar.py and
2) I would have guessed that """b. mingle test modules with the production
modules""" would be the most popular among Python developers. I like
having the tests very close to what they're testing. But are there
load-time issues? Is it considered bad form to have test code in the live
3) Related to the two above: if we do have tests and plugins, how will
those be packaged? Will we need to download two files if we want a
third-party's plugin? Or maybe plugins should have the tests inside the
file regardless of what we do throughout the rest of the code?
On Mon, October 29, 2007 7:57 pm, James G. Sack (jim) wrote:
> Summary (but please read additional text beyond the summary)
> ==> Point (1) below requires consensus and a decision. I believe I could
> manage to do and/or lead the effort[*] on the remaining points. The
> conversion would take a few days; the documentation probably longer.
> [*] except for attending to build changes that may be required.
> 1) We should create a subdir in each source directory to hold unit
> tests. The tests would look in their immediate parent directory for
> their target module. Test modules should follow the existing practice of
> appending a "-_Test" suffix to the name of the module under test.
> The subdir might be ".../tests", or ".../test" if ".../t" is thought too
> cryptic. (TBD)
> NB: The distribution build process will need to be examined, and perhaps
> adjusted to skip the test subdirs and contents.
> 2) Move the few tests we have into their respective new ".../t"
> locations, fix their names as necessary, and update per current codebase
> so that they run and succeed.
> 3) Revise the existing regression test driver RunAllTests.py to find
> tests in the source tree rather than the parallel directory (as now).
> Extend or improve options as required, and document usage.
> 4) Create wiki page for developer guidelines, tips, etc
> 5) Start writing unit test modules and developing regression testing
> Background and Further Explanation
> Gramps presently has 11 unit test modules (in 3 subdirs under .../test).
> The unit tests are run as a regression test batch via the script
> Individual tests can be run from the command line in their respective
> There is also one regular module (under .../src) containing an embedded
> unit test, that is, a unittest-based test -- run when executed as a
> program (via if __name__ == "__main__"). This should be extracted to an
> external test module.
> In addition to the tests based on the python unittest module, there
> are several other test scripts in the .../test directory. At least one
> of these (dates.sh) uses supporting test code in .../src.
> While these may be very useful, they deviate from the unit test
> philosophy of clearly indicating a success or fail result such as is
> desirable for automated testing. For now, I recommend lumping them
> into a "miscellanea" category. They can remain in the .../test dir.
> We have about 600 total python code modules, so the coverage isn't very
> significant. In addition, some of those 11 existing test files seem to
> have gotten a bit stale.
> Writing standalone unit tests is fairly easy, but making a serious
> regression test _framework_ requires a bit more planning, which I
> discuss in the two more sections below. Further discussion invited!
> Code Organization
> In my opinion, the existing deployment of test code sources in a
> parallel directory tree partially explains the underuse of unit testing.
> There are several ways of organizing test modules. The most popular ones
> seem to be
> a. put test modules in a parallel tree (as now)
> b. mingle test modules with the production modules
> c. create a test subdirectory in each production directory
> General note: Both (b) and (c) _may_ make the distribution-building
> process more complicated by requiring explicit ignores (or includes). I
> suspect it is not a big difficulty, though
> Style (a) has the major disadvantage of effectively hiding test modules
> from developer view, making it harder to discover existing test modules
> or to quickly implement a new one because of the directory navigation
> required in the editing environment. In addition, the test module has a
> somewhat cumbersome import operation for the module-under-test.
> Style (b) is preferred by many (but not me) and is the easiest for
> discovery and creation of test modules. Any "clutter" objection _might_
> be countered by the argument that test code is as important as the
> working code.
> Style (c) is _my_ preference because it segregates test modules from
> working (or production) modules, and also gives a convenient place to
> put test data and maybe other files without cluttering the production
> code. Python itself uses this test file organization. But, consider:
> - test modules are not as discoverable as in (b), but it is much easier
> than for (a) -- one need only peek into the immediate test subdir
> - test modules need a "sys.path.append('..')" to be able to import the
> module-under-test. This would not be needed with (b), but is similar to
> what is presently used in (a).
> - The test subdir name might be "./test" or "./tests", or perhaps "./t",
> the latter being the subdir name commonly used in perl source trees.
> Although it would be slightly cryptic, I have a slight preference for
> using a single letter subdir name "t", on the basis that it makes it
> less painful for developers to simultaneously edit source and test
> code. I would like to hear other opinions on this.
> ==> Recommendation: We should change to style (c), although I would
> respect community preferences for (b). We should do this before adding
> new tests. Then we should publish some guidelines encouraging unit test
> writing, and illustrating techniques and required framework interfaces
> (which are minimal, as discussed below). Objections to naming the subdir
> ".../t" are invited.
> Batch Testing
> Disclaimer: Although I am somewhat of an evangelist for unit testing
> and while I have written lots of standalone python unit tests, I am
> not an expert on managing whole trees of tests as is needed for a nice
> automated regression test capability.
> Recently, however, I have looked into the magic in our own
> .../test/RunAllTests.py and in python's test.test_support and
> test.regrtest modules. I have finally even got around to studying the
> unittest module in order to figure out how all this magic works, :-)
> It's not really as complex as a first reading of the unittest
> documentation suggests. If it is thought useful, I could probably throw
> together a wiki page describing the python unittest facility for
> developers who may be as confused as I was a while ago.
> Our existing RunAllTests.py is a reasonable starting point for a new
> utility after converting to organization style (c). The existing
> convention of naming tests with a suffix "_Test" seems reasonable given
> the abundance of modules with a leading underscore. The more common
> convention of using a prefix "test_" seems ugly and error prone when
> modules already have a leading underscore.
> Some of the existing tests have names other than module_Test.py, but I
> believe it is useful to stick to convention of naming the test after
> the module-under-test wherever possible.
> ==> Recommendation: Rewrite RunAllTests.py to search in the source tree
> for test modules *_Test.py. Modify to allow an optional target (subtree
> or single file or list?) rather than always testing the entire Gramps
> tree. Document usage. There are a couple of options needing further
> investigation -- logging and possible performance test integration.