I've been struggling a while with tests in Lib/test, originally CPython
tests, that gave different results each time I ran them. The cause in my
case appears to be a test framework problem that would affect others, so
I thought I'd share it.
In my case, it showed up in test_fileio, which came up with a shower of
"OSError: unlink(): an unknown error occurred", that seemed to get worse
the more real bugs I fixed. The problem, I believe, is simply that
tearDown() doesn't get called if the test ends by throwing an exception.
In my case, this leaves the test file open, and subsequent attempts to
unlink() (i.e. delete) the file fail. Tests are not then independent,
rather the first error is meaningful, and others can be misleading. The
file is held open by unreferenced objects, so it is sensitive to garbage
collection: single-stepping it under the debugger would often make the
problem go away entirely.
It might help if tests were designed or modified to use try-finally to
guarantee clean up. But I was surprised unittest didn't take care of
tearDown irrespective of how an individual test went. I guess CPython is
saved from this because garbage collection largely is immediate.