I'd like to propose that we switch all testing to use the nose testing
framework [1]. Almost two years ago I first broached this subject, but
there was a lot of unanswered questions and unresolved issues. Today,
that is no longer the case IMO. I have pushed a branch [2] which has
fully integrated all the existing tests into nose. I suggest reading
the documentation [3] for a full rundown of how everything works.
[1]: http://somethingaboutorange.com/mrl/projects/nose/
[2]: http://gitorious.org/python-markdown/mainline/commits/tests
[3]: http://gitorious.org/python-markdown/mainline/blobs/tests/docs/test_suite.txt
Oh, and the branch currently has one failing test which I purposely
left failing to show what the output looks like. Go ahead and play
with it.
There are a few reasons why I want to do this.
(1). The existing framework provided no way to run the UnitTests and
DocTests which have been added over the last couple years. Nose
discovers and runs these tests automatically. We are starting to see
more extensions developed by the community. Those UnitTests are
necessary to make sure we don't break other peoples extensions - so we
need to be running them regularly.
(2). Any time a new set of settings are required for a test, the
actual code for our testing framework needs to be edited. Currently,
safe_mode and output_format are defined explicitly, while extensions
are defined by the directory name. And there is no way to pass in or
test different sets of config options for extensions.
(3). It is currently not easy to run tests from other implementations
through our existing framework. If we want to match perl/php
implementations, we should be able to run their tests.
(4). I have found it very annoying that each test dir generates a
separate html report - especially when many extensions only contain
one test each. Why can't all the tests from all dirs be written to one
file? I'm also not crazy about how the existing html reports are
formatted. Why aren't the results inline with the list of tests
(instead of at the bottom) and why aren't we using unified diffs?
(5). I've come across a few bugs recently that the existing framework
doesn't even allow us to check for. For example, the current framework
does some weird stuff with leading whitespace that hid the problem
reported in [ticket 44].
[ticket 44]: http://www.freewisdom.org/projects/python-markdown/Tickets/000044
Now, I realize the existing framework could be refactored to address
all of these issues, but is was actually less work to make nose give
us everything we already have and more. What I have in the branch
addresses every one of the above issues. And, as each txt/html file
pair is in it's own UnitTest, each is sandboxed from the rest.
Plus, we can use things like nose's coverage plugin (which, btw,
indicates that some areas of Markdown are seriously undertested). We
can only run tests that failed (or passed) on the last run. We can run
any small subset of tests at a time. All these things come with no
work from us. Nose gives it to us for free. We lose nothing and have
much to gain.
Still not convinced? Go check out this [video] of a Pycon 2009 Panel:
"Functional Testing Tools in Python". A question asked at 46:19
sparked a little rant about projects that implement their own testing
frameworks. Some valid arguments are provided. These are the guys who
write the testing frameworks. They say we should leave the code that
runs tests to them. After putting together the nose plugins that make
nose work for us, I have to agree. I'd rather maintain those two
plugins than have to continue maintaining our own framework.
[video]: http://blip.tv/file/1947342
Personally, I'm likely to use this from now on. Even if this doesn't
get into the master branch, I will continue to develop on my tests
branch and cherrypick commits to merge back to master. Obviously, I'd
rather not need to do the cherrypicking.
--
----
\X/ /-\ `/ |_ /-\ |\|
Waylan Limberg
|
