[Pyunit-interest] testing order of classes in a module

[Andrew D. <ad...@mi...>]

I'm writing an interface to a web service.  Because each request
takes a while (a couple of second) I wrote a wrapper which
catches all calls and results and saves them to a file.  In that
way I can run my unit tests against the file, which is much
faster.  (I retest against the main server too, but not as often.)

The server is stateful, so I must do the same calls in the same
order.  I assumed the order the tests were executed in a module
are always the same, so I simply pickled all socket input and
output to a file, expecting I could read the picked data out
in the same order it was put in.

This doesn't appear to be the case.  One of my modules fails
a lot.  I can build the log file from live data, then run it
again some time later and it fails.  All without me changing
the code.

Looking at the data suggests that the tests order changes.

Looking at the code I think I pinned it down to
TestLoader.loadTestsFromModule.  It gets the list of test
cases from the module but doesn't sort them.  Here's my patch

*** /usr/local/lib/python2.3/unittest.py        Thu Jan  2 13:31:43 2003
--- unittest.py Sat Jan 11 01:12:41 2003
***************
*** 447,458 ****

      def loadTestsFromModule(self, module):
          """Return a suite of all tests cases contained in the given module"""
!         tests = []
          for name in dir(module):
              obj = getattr(module, name)
              if (isinstance(obj, (type, types.ClassType)) and
                  issubclass(obj, TestCase)):
!                 tests.append(self.loadTestsFromTestCase(obj))
          return self.suiteClass(tests)

      def loadTestsFromName(self, name, module=None):
--- 447,460 ----

      def loadTestsFromModule(self, module):
          """Return a suite of all tests cases contained in the given module"""
!         klasses = []
          for name in dir(module):
              obj = getattr(module, name)
              if (isinstance(obj, (type, types.ClassType)) and
                  issubclass(obj, TestCase)):
!                 klasses.append(obj)
!         klasses.sort(lambda x, y: cmp(x.__name__, y.__name__))
!         tests = map(self.loadTestsFromTestCase, klasses)
          return self.suiteClass(tests)

      def loadTestsFromName(self, name, module=None):


Is my analysis correct?  Is this an appropriate solution?

I do know that with proper unit tests I am supposed to be able
pull each one out and test them in any order.    I could write
the scaffolding to do that, but it's tedious.  The above just
works, so seems to me to be the right solution.

I see elsewhere mention of the sort function to use, so the
better solution might be

          klasses.sort(lambda x, y:
             self.sortTestMethodsUsing(x.__name__, y.__name__))

but these aren't methods, so I'll defer the decision to you all.

					Andrew
					da...@da...