Re: [Pyunit-interest] testing order of classes in a module

[Andrew D. <ad...@mi...>]

Steve Purcell:
> The framework relies on the test author making sure that individual test case 
> do not interfere with each other's results. In your case, sorting the test 
> cases by class name solves the problem, but I think the real difficulty is 
> how you have defined the tests.

Yes, I think I do understand that philosophy.

> If your current test cases interfere with each other due to the stateful 
> nature of the server, you should probably do one of the following things. 
> Firstly, you could arrange that the server is reset after every test case by 
> overriding the 'tearDown()' method of TestCase. Or secondly, you could merge 
> interdependent test cases into single test cases, such that they run 
> together.

I have no control of the server, so I can't do the first case.  I could
merge the test cases together, but that doesn't feel right.  I would be
mixing quite different tests together in a test case.

What I have are a set of independent test cases.  If I run them in
"live" mode, I do not have a problem, no matter the ordering of
the test cases, nor even the order of the method calls in a test case.
(I say that last because I noticed that unittest does sort the methods
by name, so there is some enforcement of order -- just not one that
matters to me.)

On thinking about this, I realized the statefulness of the server
really wasn't the problem.  Consider the following tests, which
is characteristic of what I'm doing.

import unittest, testsupport, sys, os, urllib2
opener = None
PICKLE_FILENAME = "test_websites.pickle"

class TestPythonOrg(unittest.TestCase):
  def testMainPage(self):
    infile = opener("http://www.python.org")
    .... check for items on that page
  def testDocsPage(self):
    infile = opener("http://www.python.org/doc/")
    .... check for items on that page


# Make sure the first hit from a goole search has
# the the search text in the description.  Totally
# fake API.
class TestGoogle(unittest.TestCase):
  def testSimpleSearch(self):
    form1 = parse_form(opener("http://www.google.com/")
    form1["q"] = "python"
    infile = form1.submit(opener = opener)
    results = parse_google_html_stream(infile)
    assert "spam" in results.group[0].text.lower()

if __name__ == "__main__":
  s = os.environ.get("TEST_SOURCE", "PICKLE")
  if s == "PICKLE":
    opener = testsupport.OpenFromPickle(PICKLE_FILENAME)
  elif s == "LIVE":
    opener = urllib2.build_opener()
  elif s == "CREATE":
    opener = testsupport.CreatePickle(PICKLE_FILENAME)
  else:
    raise TypeError("Unknown TEST_SOURCE: %r" % (s,))
  unittest.main()

Notice that every single test is independent of the others.
Inside of a test there is state.

Now, it may be that some of these tests take a non-trivial
amount of time to run -- perhaps 10 seconds.  These
add up, and it's part of the unittest testing philosophy
that these tests be fast.  (as in
   http://c2.com/cgi/wiki?OptimizingUnitTests )
I can agree with this.  Getting validation in 5 seconds
total as compared to 2 minutes really makes a difference.

So what I did was provide something else which implements
the URL opener interface.  This reads the requests from
a local file instead of calling the server.  It's a lot
faster, but the implementation assumes that all the methods
are always called in the same order.

What I want could work with the unittest interface if
I told each method how to get the stored data for that
specific method.  Eg, here's one possible API

   def testSimpleSearch(self):
     opener = picklestore.fetch_for("testSimpleSearch")
     form1 = opener("http://www.google.com/")
     ...

if __name__ == "__main__":
    if "PICKLE", picklestore = support.ReadPickleStore()
    if "LIVE", picklestore = support.NoPickleStore()
    if "CREATE", picklestore = support.CreatePickleStore()

However, that makes the code a bit more complicated,
since not only do I need that extra line, but I have to
remember to name each test differently, even across all
unit test instances.  (I considered inspecting the stack
to get the method and class name automatically, but that
didn't work because some of the methods are shared in a
base class.)

There's some minor implementation annoyances too.  Eg,
I would either need random access to a named pickle string
from a (akin to what the dbms give me .. hmmm..) or I
need to read the whole data set into an in-memory dict.
(only about 200K, so I guess that isn't too bad either.)

Rather than do all that, I decided to try the simplest
thing possible, and make the test cases be ordered.  That
was about 5-10 minutes vs. an hour or more of coding.

					Andrew
					da...@da...