Re: [Jython-dev] state of regrtests

[Jim B. <jim...@py...>]

Keeping a stable regrtest is something we continuously need to work on. But
it's not at all easy, especially on Windows. The wrap up of the 2.7.0 dev
cycle saw *significant* time spent on regrtest failures.

The biggest issue currently is choice of Java version. Java 7 works better
with our regrtests. Java 8 complains much more. Consider the craziness we
do in test_sort, which raises exceptions in test_sort due to greater
restrictions on sorting, such as Java 8 mandating the comparator be well
defined. Putting on my triage hat, so to speak: let's focus on building and
testing against Java 7, given we have bigger problems to fix, at least now
for 2.7.1 and especially given that we are planning a release candidate on *Nov
5*. But definitely something we need to put time into for 2.7.2. Fixing
these tests (most likely) or underlying bugs in the runtime is especially
important for Java 9, a focus of the 2.7.2 release. (Java 9 is surely not
going to make things easier for us after all.) Perhaps some skips should be
conditioned on the Java version.

The second biggest issue is tests that don't properly clean up. It is still
noticeable when the test are run on Windows, between the lack of
deterministic collection and the inability to remove files that are in use
(Windows specific). We have done some work here, but more needs to be done.

Lastly we have networking tests that by their nature are nondeterministic
and can also fail because of running in specific network environments, such
as starting up a VPN or running in corporate environments. Maybe do all
your testing at Starbucks? ;) It's possible we can do better skips.

on OS X 10.11, Java 7 (1.7.0_75-b13)

     [exec] 3 fails unexpected:
     [exec]     test_classpathimporter test_select test_sys_jy

I then rerun any failed tests to see if flaky or not, with

dist/bin/jython regrtest.py --verbose test_classpathimporter test_select
test_sys_jy

At which point only test_classpathimporter fails (usually). There's an open
bug on test_classpathimporter, but for Windows only (
http://bugs.jython.org/issue2309). We need to look into this. In any event,
rerunning just the failed tests, individually as necessary, makes sure we
are not having flaky tests obscure important regressions.

On Ubuntu 15.04 (I will try 15.10 in a few days...), I get the following:

     [exec] 4 fails unexpected:
     [exec]     test_classpathimporter test_jython_launcher test_select
     [exec]     test_sys_jy

Retrying, I observe test_jython_launcher is problematic on Ubuntu 15.04, in
addition to test_classpathimporter. We will want to look into this failure.

- Jim

On Wed, Oct 21, 2015 at 8:28 PM, Stefan Richthofer <Ste...@gm...
> wrote:

> Hello everybody,
>
> on my last commit I missed an issue at first, which was mainly due to the
> overwhelming
> number of routinely failing regrtests.
> regrtests are actually not meant to fail, are they? Correct me, if I'm
> wrong, but our
> workflow should be:
>
> 1) fix an issue
> 2) check if regrtests pass
> 3) no: Fix it; goto 2)
>    yes: commit
>
>
> However, the workflow actually is:
>
> 1) fix an issue
> 2) note that so many regrtests fail that you can hardly assess whether you
> caused some of this
> 3) obtain another clone of Jython in the state before your fix
> 4) run old regrtests
> 5) try to compare results before your fix and after
> 6) note that it is actually not deterministic which regrtests fail
> 7) run again and again to get a feeling for the whole set of tests that
> potentially fail from time to time
> 8) Note regarding 7: Roughly half the runs hang, e.g. with [exec] error:
> [Errno 24] Cannot allocate thread pool for server socket
>    In that case you must start over and it takes another 20 minutes to get
> another sample of resulting failures
> 9) try to assess whether there are tests that fail significantly more
> often after your fix than before
> 10) check whether some of these might be caused by your change
> ...
> ?) finally merge and hope you got it right
>
>
> I remember that we had the number of failing regrtests down to 4-6 in
> 2.7.0., which was still not ideal,
> but at least somehow trackable. Now I get 22-26 failing regrtests and lots
> of hangings by current Jython
> repository version. E.g. one of my best runs this evening/night resulted
> in:
>
>      [exec] 22 tests failed:
>      [exec]     test_classpathimporter test_cmd_line test_grp
> test_httpservers
>      [exec]     test_jython_launcher test_logging test_mailbox test_os
> test_os_jy
>      [exec]     test_posix test_posixpath test_pwd test_quopri test_shutil
>      [exec]     test_site test_site_jy test_subprocess test_subprocess_jy
>      [exec]     test_sys_jy test_tarfile test_weakset test_zipimport_jy
>
>
> Seriously, these are almost 500% more failing regrtests than in 2.7.0.
> This way regrtesting
> is basically unusable for its intended purpose. Any suggestions how to
> improve this? Can we
> maybe please just remove routinely failing regrtests and make them
> explicit issues? To get
> back to the situation that regrtests don't fail per default?
> I'm not expert enough about every Jython detail to tell which of these
> tests fail for good
> reasons, so this is something the community must agree on I suppose. And
> please let's solve
> this before 2.7.1 release.
>
> Best
>
> Stefan
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Jython-dev mailing list
> Jyt...@li...
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>