[Jython-dev] state of regrtests

[Stefan R. <Ste...@gm...>]

Hello everybody,

on my last commit I missed an issue at first, which was mainly due to the overwhelming
number of routinely failing regrtests.
regrtests are actually not meant to fail, are they? Correct me, if I'm wrong, but our
workflow should be:

1) fix an issue
2) check if regrtests pass
3) no: Fix it; goto 2)
   yes: commit


However, the workflow actually is:

1) fix an issue
2) note that so many regrtests fail that you can hardly assess whether you caused some of this
3) obtain another clone of Jython in the state before your fix
4) run old regrtests
5) try to compare results before your fix and after
6) note that it is actually not deterministic which regrtests fail
7) run again and again to get a feeling for the whole set of tests that potentially fail from time to time
8) Note regarding 7: Roughly half the runs hang, e.g. with [exec] error: [Errno 24] Cannot allocate thread pool for server socket
   In that case you must start over and it takes another 20 minutes to get another sample of resulting failures
9) try to assess whether there are tests that fail significantly more often after your fix than before
10) check whether some of these might be caused by your change
...
?) finally merge and hope you got it right


I remember that we had the number of failing regrtests down to 4-6 in 2.7.0., which was still not ideal,
but at least somehow trackable. Now I get 22-26 failing regrtests and lots of hangings by current Jython
repository version. E.g. one of my best runs this evening/night resulted in:

     [exec] 22 tests failed:
     [exec]     test_classpathimporter test_cmd_line test_grp test_httpservers
     [exec]     test_jython_launcher test_logging test_mailbox test_os test_os_jy
     [exec]     test_posix test_posixpath test_pwd test_quopri test_shutil
     [exec]     test_site test_site_jy test_subprocess test_subprocess_jy
     [exec]     test_sys_jy test_tarfile test_weakset test_zipimport_jy


Seriously, these are almost 500% more failing regrtests than in 2.7.0. This way regrtesting
is basically unusable for its intended purpose. Any suggestions how to improve this? Can we
maybe please just remove routinely failing regrtests and make them explicit issues? To get
back to the situation that regrtests don't fail per default?
I'm not expert enough about every Jython detail to tell which of these tests fail for good
reasons, so this is something the community must agree on I suppose. And please let's solve
this before 2.7.1 release.

Best

Stefan