On Tuesday 30 October 2007, James G. Sack (jim) wrote:
> Brian Matherly wrote:
> > Thanks for stepping up for this task. You are certainly welcome to
> > tackle it.
> > I have a few questions/concerns:
> Ahh, great questions! And a chance to preach :-)
> > 1) What exactly is your end goal for unit tests?
> - unit test writing probably can't be "forced", but any testing is
> better than no testing
> - once a developer discovers the benefits of unit tests, it is (I
> believe) rare for them to abandon their use
> - if a good test exists, it is *much* less traumatic to attempt bug
> fixing and feature addition. "good" means it verifies correct behavior
> in the vicinity of the code needing change.
> - ("Fram Oil Filter rule") You can test it now or test it in
> integration, or acceptance, or when it breaks; you cannot avoid testing,
> so it is best to write the test along with the code (when [you think]
> you know what the code is supposed to do). Be aware that test code
> sometimes adds up to a significant amount of code.
And maintaining tests can take more time than maintaining the code it's
testing (due to needed setup, good enough parameter coverage and result
> - putting the tests under source control ensures they are there when a
> new bug is discovered
> - if you ain't tested it, you shouldn't advertise having it
> > 2) What things do you hope to prove with automated unit tests?
> - automated tests require a commitment to keeping tests current, so they
> prove _very little_ without ongoing maintenance of tests along with the
> working code. But to the extent that it does work, one can adopt various
> strategies, ranging from continuous testing to release-time checking. it
> probably takes a while to mature, but my hope is that unit test
> practices will eventually become considered so important that no module
> that has an associated unit test ever gets checked in unless the unit
> test (is updated as required) and passes. I do know (good) programmers
> who refuse to modify code that does not have any (or adequate) unit
> tests. - testing of individual subtrees may still be useful even if the
> full tree testing gets broken
> > 3) What code should have unit tests?
> - anything that can produce a OK/fail condition.
> - the most common test I have written is to check that a function
> returns the expected output(s) for some given input(s).
> - anything that is a demonstrated bug; a bug is just some behavior that
> was never tested for. Add a test to demonstrate the bug, fix the bug,
> show that it's now fixed, have a permanent test that pops up if the bug
> ever returns.
> - especially common is to verify behavior at all input limits, and to
> verify that all exceptions are raised when they should be
Tests are better only in case the tests actually reduce the work (in
medium/long term). There's no point in wasting resources for writing,
maintaining and running a test unless it finds a defect. Currently Gramps
has nearly 150 KLOCs of code, there's no point in doubling that maintenance
burden. (Just running tests for all functions there would take a lot of
So, I'd like some pre-thought on what should be tested / offer best ROI:
- Functionality that could cause data corruption or loss
- Important internal functionality which breakage is hard to catch
- Functionality that covers/exercises a lot of the underlying (or above)
code/functionality with little test-code and/or need for maintenance
- Stable internal APIs used by multiple components
- New code
> > 4) What code should not have unit tests?
Functionality that is very visible to user might not be a priority because
as users find faults in that easily, so test don't (necessarily) need to.
> I'm not quite sure, but...
> - it is more difficult to test interactive code
Based on what I've understood, testing is easier, at least if you compare
the needed effort against the test coverage. If I remember correctly the
comments on the mailing list, at least half Gramps code deals with UI, so
you might not get very good coverage without this.
The hard thing is that it doesn't point out as well where the problem is.
So you need internal API testing also.
> - gui testing is different from anything I have done via code, but I
> understand that gui testing tools are becoming more readily available
> and useful
Gtk offers accessibility APIs for introspecting the UI contents and invoking
> - code that operates on complicated state that is hard to separate into
> simple tests will be .. hard! Interestingly, code that is hard to test
> often stimulates rethinking the code design and that frequently results
> in code architectural improvements </end hand waving>
> > 5) I presume you don't intend to one day have a unit test for every
> > line of code in Gramps.
> - that _would_ be pretty ambitious -- I don't expect to live that long
> <heh>. But some projects do aim to test every functional behavior, and
> some projects do try to measure code coverage. So the goal is not that
> outlandish. Realistically though, testing doesn't really guaranty
> - python has only about half as many test_XXX.py files as non test files
> int their library (builtin) code. Perl6 developers claim to have 100K
> test cases that they say will "define what perl6" is.
> > 6) Who will maintain the unit tests as the code changes?
> - that's a key question. If the tests are not visible, no one and they
> might as well not exist. If that were the prospect, I would write my own
> unit tests on code I worked on, and keep it to myself. If unit test code
> modules are visible and start out working, there's a potential that
> developers will use them to verify that their bugfixing and feature
> addition hasn't broken anything. If maintenance mods render a test (or
> more commonly a part of a test module) inapplicable or wrong, the
> developer _should_ remove or update the test. This is the part that does
> not happen without some effort. Encouragement helps. Making test writing
> easy helps. Having a test update party before release might help. Best
> is if developers decide on their own that tests help them write good
> > 7) I would hate to make unit testing a policy as it might discourage
> > new developers. I can see the response now: "Dear Joe, your patch for
> > the new xyz feature is great. But I can't commit it because it doesn't
> > have a unit test. Please write a unit test and resubmit your patch."
> - I agree that OSS projects can't very well implement such policies by
> fiat. It probably has to be a botttom up and team-spirit effect,
> although it can be encouraged at project management level. I do believe
> a few good code examples and a few enthusiasts can stimulate good
> results. - I would also repeat that any unit testing is better than no
> testing - if a patch breaks an existing test, it may be easy for a
> reviewer to update the test in the process of checking the patch. If not,
> maybe the patch is too big and should be partitioned? If it comes to it,
> then the test may have to be cut down or even discarded if it no longer
> provides any benefit.
> > All in all, I personally prefer option "a" because it is the least
> > obtrusive and the least commitment. We don't have strict policies
> > around here about unit test procedures. And I'm not sure that we are
> > interested in implementing any. If the tests become stale, someone
> > could pick them up in the future.
> I've already stated my opinion that "a" (which we have now) is a major
> reason that developers ignore existing tests and which discourages
> writing new ones. I would like to be able to convince people that "b" or
> "C" is more likely to get existing tests used and maintained and
> hopefully encourage creation of new tests. I like "c" best, but know
> some very respected people who like "b".
> > Option "c" is tolerable for me. But I will be quite annoyed if you
> > litter the source with subdirectories named "t" and then get tired of
> > Gramps and move on. The contents of those directories will one day grow
> > stale and we'll have to remove them by hand.
> - sure, svn makes it pretty easy to delete things. I would say that
> empty directories do no good, but that as soon as a test is deemed
> necessary, the creator should go ahead and create the subdir if not
> > Thanks again for your willingness to contribute. Please don't be
> > discouraged by my questions...
> No, questions are good! challenging questions are better!
> >..I like unit tests as much as the next guy...
> Me, more than the next guy, I guess. Bottom line is there is no silver
> bullet in any methodology or any specific practice.
> >..But in my experience, everyone is quick to say "yeah, we should do
> > unit tests", but are slow in actually writing and maintaining them..
> My answer to this is give it a try for a while. Maybe I might be the
> only one who does anything with it at first. I would hope to demonstrate
> with even some small efforts that there are very noticeable benefits. If
> I fail to make any impact or measurable contribution, well, then I will
> probably get chased away or become discouraged, and you can always
> delete the tests with not much work.
> >..(myself included). I hope you are prepared to run the testing project
> solo if people don't put their code where their mouth is.
> Well, I do _intend_ to stick around a while. And I'm am prepared to
> nibble away at test creation (and maintenance) as I contribute to
> mainline working code. I'm retired and don't plan on going anywhere for
> a while so there's no major obstacle to putting in some time.
> Really good questions. Hope my answers were the kind you were looking.
> You may have to (learn to) excuse my rambling; that's not unusual for
> me. ;-)
> Other developers should jump in too. We could use some more opinions,
> even if they are expressing reservations or even outright doubts.
> In the meantime, I am working on an update to the RunAllTests.py driver,
> and a demo of a small unit test development example on real code. It
> would be nice to have some consensus by (say) a week from now so that I
> might start some initial checkins along these lines.
> Cheers, and Regards,