jython-dev Mailing List for Jython (Page 51)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>
<div>It appears for me not to be a Java8 vs Java7 issue. I tested with both and roughly get the same result.</div>

<div>I wouldn&#39;t have started this, if it were only test_sort and a hand-full of others failing just on Java8.</div>

<div>&nbsp;</div>

<div>Recent run:</div>

<div>&nbsp;</div>

<div>Java8:</div>

<div>&nbsp;&nbsp;&nbsp;&nbsp; [exec] 26 tests failed:<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_classpathimporter test_cmd_line test_grp test_httpservers<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_jython_launcher test_list_jy test_logging test_mailbox<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_marshal test_os test_os_jy test_posix test_posixpath test_pwd<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_quopri test_shutil test_site test_site_jy test_socket<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_sort test_subprocess test_subprocess_jy test_sys_jy<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_tarfile test_threading test_zipimport_jy</div>

<div>&nbsp;</div>

<div>Java7:</div>

<div>&nbsp;&nbsp;&nbsp;&nbsp; [exec] 23 tests failed:<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_classpathimporter test_cmd_line test_grp test_httpservers<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_jython_launcher test_logging test_mailbox test_os test_os_jy<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_posix test_posixpath test_pwd test_quopri test_shutil<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_site test_site_jy test_subprocess test_subprocess_jy<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_sys_jy test_tarfile test_threading test_weakset<br/>
&nbsp;&nbsp;&nbsp;&nbsp; [exec]&nbsp;&nbsp;&nbsp;&nbsp; test_zipimport_jy</div>

<div>&nbsp;</div>

<div>I also considered it might be due to using standalone, which I often do, but without standalone it is also</div>

<div>the same situation (the posted runs above were done after an ordinary build). I admit that some of the</div>

<div>failing tests seem to work fine when run on their own rather than with ant regrtest, e.g. test_zipimport_jy</div>

<div>appears to work fine then. Still - in that case it is an issue with ant regrtest or something - not exactly satisfying!</div>

<div>test_weakset sometimes passes on its own, sometimes fails with e.g.</div>

<div>File &quot;Lib/test/test_weakset.py&quot;, line 429, in test_weak_destroy_and_mutate_while_iterating<br/>
&nbsp;&nbsp;&nbsp; self.assertEqual(len(s), len(t))<br/>
AssertionError: 53 != 50</div>

<div>For me this looks not like an obvious or trivial issue caused by a wrong flag or something.</div>

<div>This was just a random sample and would need to be investigated systematically.</div>

<div>&nbsp;</div>

<div>@Jim It is interesting that such few tests fail on your system.</div>

<div>&nbsp;</div>

<div>However it is no good situation to have such a gap between different systems</div>

<div>(especially if it&#39;s the same platform).</div>

<div>I remember from PyCon sprints that Alex also had some more tests failing.</div>

<div>So I am curious whether my system is the exotic with &gt;20 tests failing.</div>

<div>Would be good to hear some statistics from others about this!</div>

<div>&nbsp;</div>

<div>How should I procceed with the failing tests? I could go through them, run them in verbose mode and</div>

<div>create issues as far as they don&#39;t yet exist. If corresponding issues already exist in the tracker, the test</div>

<div>should be moved to expected failures, if the solution is long-term-pending, shouldn&#39;t it?</div>

<div>Cleaning up these tests is some work, so I would much appreciate if someone who also experiences more</div>

<div>issues than Jim could help on this. I will also repeat the tests on other system as soon as I find time and</div>

<div>opportunity for this.</div>

<div>&nbsp;</div>

<div>-Stefan</div>

<div>
<div>&nbsp;</div>

<div>&nbsp;</div>

<div style="margin: 10.0px 5.0px 5.0px 10.0px;padding: 10.0px 0 10.0px 10.0px;border-left: 2.0px solid rgb(195,217,229);">
<div style="margin: 0 0 10.0px 0;"><b>Gesendet:</b>&nbsp;Donnerstag, 22. Oktober 2015 um 23:00 Uhr<br/>
<b>Von:</b>&nbsp;&quot;Jim Baker&quot; &lt;jim...@py...&gt;<br/>
<b>An:</b>&nbsp;&quot;Stefan Richthofer&quot; &lt;Ste...@gm...&gt;<br/>
<b>Cc:</b>&nbsp;&quot;Jython Developers&quot; &lt;jyt...@li...&gt;<br/>
<b>Betreff:</b>&nbsp;Re: [Jython-dev] state of regrtests</div>

<div>
<div>Keeping a stable regrtest is something we continuously need to work on. But it&#39;s not at all easy, especially on Windows. The wrap up of the 2.7.0 dev cycle saw <b>significant</b> time spent on regrtest failures.

<div>&nbsp;</div>

<div>The biggest issue currently is choice of Java version. Java 7 works better with our regrtests. Java 8 complains much more. Consider the craziness we do in test_sort, which raises exceptions in test_sort due to greater restrictions on sorting, such as Java 8 mandating the comparator be well defined. Putting on my triage hat, so to speak: let&#39;s focus on building and testing against Java 7, given we have bigger problems to fix, at least now for 2.7.1 and especially given that we are planning a release candidate on <b>Nov 5</b>. But definitely something we need to put time into for 2.7.2. Fixing these tests (most likely) or underlying bugs in the runtime is especially important for Java 9, a focus of the 2.7.2 release. (Java 9 is surely not going to make things easier for us after all.) Perhaps some skips should be conditioned on the Java version.</div>

<div>&nbsp;</div>

<div>The second biggest issue is tests that don&#39;t properly clean up. It is still noticeable when the test are run on Windows, between the lack of deterministic collection and the inability to remove files that are in use (Windows specific). We have done some work here, but more needs to be done.</div>

<div><br/>
Lastly we have networking tests that by their nature are nondeterministic and can also fail because of running in specific network environments, such as starting up a VPN or running in corporate environments. Maybe do all your testing at Starbucks? ;) It&#39;s possible we can do better skips.</div>

<div>&nbsp;</div>

<div>
<div>on OS X 10.11, Java 7 (1.7.0_75-b13)</div>

<div>&nbsp;</div>

<div>
<div>&nbsp; &nbsp; &nbsp;[exec] 3 fails unexpected:</div>

<div>&nbsp; &nbsp; &nbsp;[exec] &nbsp; &nbsp; test_classpathimporter test_select test_sys_jy</div>

<div>&nbsp;</div>

<div>I then rerun any failed tests to see if flaky or not, with&nbsp;</div>

<div>&nbsp;</div>

<div>dist/bin/jython regrtest.py --verbose test_classpathimporter test_select test_sys_jy</div>

<div>&nbsp;</div>

<div>At which point only test_classpathimporter fails (usually). There&#39;s an open bug on test_classpathimporter, but for Windows only (<a href="http://bugs.jython.org/issue2309" target="_blank">http://bugs.jython.org/issue2309</a>). We need to look into this. In any event, rerunning just the failed tests, individually as necessary, makes sure we are not having flaky tests obscure important regressions.</div>

<div>&nbsp;</div>

<div>On Ubuntu 15.04 (I will try 15.10 in a few days...), I get the following:</div>

<div>&nbsp;</div>

<div>
<div>&nbsp; &nbsp; &nbsp;[exec] 4 fails unexpected:</div>

<div>&nbsp; &nbsp; &nbsp;[exec] &nbsp; &nbsp;&nbsp;test_classpathimporter&nbsp;test_jython_launcher test_select</div>

<div>&nbsp; &nbsp; &nbsp;[exec] &nbsp; &nbsp; test_sys_jy</div>
</div>

<div>&nbsp;</div>

<div>Retrying, I observe test_jython_launcher is problematic on Ubuntu 15.04, in addition to test_classpathimporter. We will want to look into this failure.</div>

<div>&nbsp;</div>

<div>
<div>- Jim</div>
</div>
</div>
</div>
</div>

<div class="gmail_extra">&nbsp;
<div class="gmail_quote">On Wed, Oct 21, 2015 at 8:28 PM, Stefan Richthofer <span>&lt;<a>Ste...@gm...</a>&gt;</span> wrote:

<blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;">Hello everybody,<br/>
<br/>
on my last commit I missed an issue at first, which was mainly due to the overwhelming<br/>
number of routinely failing regrtests.<br/>
regrtests are actually not meant to fail, are they? Correct me, if I&#39;m wrong, but our<br/>
workflow should be:<br/>
<br/>
1) fix an issue<br/>
2) check if regrtests pass<br/>
3) no: Fix it; goto 2)<br/>
&nbsp; &nbsp;yes: commit<br/>
<br/>
<br/>
However, the workflow actually is:<br/>
<br/>
1) fix an issue<br/>
2) note that so many regrtests fail that you can hardly assess whether you caused some of this<br/>
3) obtain another clone of Jython in the state before your fix<br/>
4) run old regrtests<br/>
5) try to compare results before your fix and after<br/>
6) note that it is actually not deterministic which regrtests fail<br/>
7) run again and again to get a feeling for the whole set of tests that potentially fail from time to time<br/>
8) Note regarding 7: Roughly half the runs hang, e.g. with [exec] error: [Errno 24] Cannot allocate thread pool for server socket<br/>
&nbsp; &nbsp;In that case you must start over and it takes another 20 minutes to get another sample of resulting failures<br/>
9) try to assess whether there are tests that fail significantly more often after your fix than before<br/>
10) check whether some of these might be caused by your change<br/>
...<br/>
?) finally merge and hope you got it right<br/>
<br/>
<br/>
I remember that we had the number of failing regrtests down to 4-6 in 2.7.0., which was still not ideal,<br/>
but at least somehow trackable. Now I get 22-26 failing regrtests and lots of hangings by current Jython<br/>
repository version. E.g. one of my best runs this evening/night resulted in:<br/>
<br/>
&nbsp; &nbsp; &nbsp;[exec] 22 tests failed:<br/>
&nbsp; &nbsp; &nbsp;[exec]&nbsp; &nbsp; &nbsp;test_classpathimporter test_cmd_line test_grp test_httpservers<br/>
&nbsp; &nbsp; &nbsp;[exec]&nbsp; &nbsp; &nbsp;test_jython_launcher test_logging test_mailbox test_os test_os_jy<br/>
&nbsp; &nbsp; &nbsp;[exec]&nbsp; &nbsp; &nbsp;test_posix test_posixpath test_pwd test_quopri test_shutil<br/>
&nbsp; &nbsp; &nbsp;[exec]&nbsp; &nbsp; &nbsp;test_site test_site_jy test_subprocess test_subprocess_jy<br/>
&nbsp; &nbsp; &nbsp;[exec]&nbsp; &nbsp; &nbsp;test_sys_jy test_tarfile test_weakset test_zipimport_jy<br/>
<br/>
<br/>
Seriously, these are almost 500% more failing regrtests than in 2.7.0. This way regrtesting<br/>
is basically unusable for its intended purpose. Any suggestions how to improve this? Can we<br/>
maybe please just remove routinely failing regrtests and make them explicit issues? To get<br/>
back to the situation that regrtests don&#39;t fail per default?<br/>
I&#39;m not expert enough about every Jython detail to tell which of these tests fail for good<br/>
reasons, so this is something the community must agree on I suppose. And please let&#39;s solve<br/>
this before 2.7.1 release.<br/>
<br/>
Best<br/>
<br/>
Stefan<br/>
<br/>
------------------------------------------------------------------------------<br/>
_______________________________________________<br/>
Jython-dev mailing list<br/>
<a>Jyt...@li...</a><br/>
<a href="https://lists.sourceforge.net/lists/listinfo/jython-dev" target="_blank">https://lists.sourceforge.net/lists/listinfo/jython-dev</a></blockquote>
</div>
</div>
</div>
</div>
</div>
</div></div></body></html>

2000	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct (38)	Nov (98)	Dec (58)
2001	Jan (114)	Feb (123)	Mar (96)	Apr (66)	May (84)	Jun (72)	Jul (128)	Aug (126)	Sep (82)	Oct (80)	Nov (148)	Dec (55)
2002	Jan (137)	Feb (85)	Mar (118)	Apr (67)	May (71)	Jun (28)	Jul (69)	Aug (48)	Sep (83)	Oct (79)	Nov (54)	Dec (32)
2003	Jan (44)	Feb (47)	Mar (59)	Apr (57)	May (43)	Jun (45)	Jul (44)	Aug (39)	Sep (27)	Oct (62)	Nov (17)	Dec (23)
2004	Jan (41)	Feb (51)	Mar (38)	Apr (30)	May (25)	Jun (12)	Jul (11)	Aug (27)	Sep (16)	Oct (56)	Nov (23)	Dec (29)
2005	Jan (75)	Feb (82)	Mar (50)	Apr (77)	May (19)	Jun (104)	Jul (47)	Aug (42)	Sep (28)	Oct (143)	Nov (62)	Dec (13)
2006	Jan (20)	Feb (10)	Mar (59)	Apr (45)	May (25)	Jun (129)	Jul (162)	Aug (91)	Sep (15)	Oct (39)	Nov (186)	Dec (191)
2007	Jan (134)	Feb (140)	Mar (106)	Apr (77)	May (92)	Jun (63)	Jul (233)	Aug (102)	Sep (119)	Oct (63)	Nov (68)	Dec (32)
2008	Jan (69)	Feb (91)	Mar (129)	Apr (44)	May (18)	Jun (53)	Jul (50)	Aug (25)	Sep (11)	Oct (28)	Nov (67)	Dec (36)
2009	Jan (20)	Feb (24)	Mar (66)	Apr (53)	May (48)	Jun (48)	Jul (59)	Aug (82)	Sep (49)	Oct (30)	Nov (16)	Dec (16)
2010	Jan (52)	Feb (25)	Mar (36)	Apr (34)	May (14)	Jun (15)	Jul (14)	Aug (16)	Sep (23)	Oct (6)	Nov (4)	Dec (5)
2011	Jan (4)	Feb (22)	Mar (45)	Apr (9)	May (8)	Jun (13)	Jul (12)	Aug (4)	Sep (6)	Oct (10)	Nov (21)	Dec (5)
2012	Jan (6)	Feb (9)	Mar (25)	Apr (6)	May (4)	Jun (23)	Jul (6)	Aug (18)	Sep (21)	Oct (34)	Nov (19)	Dec (25)
2013	Jan (8)	Feb (34)	Mar (35)	Apr (4)	May (11)	Jun (4)	Jul (7)	Aug (5)	Sep (20)	Oct (12)	Nov (11)	Dec (7)
2014	Jan (10)	Feb (18)	Mar (50)	Apr (26)	May (53)	Jun (21)	Jul (12)	Aug (39)	Sep (43)	Oct (26)	Nov (8)	Dec (6)
2015	Jan (18)	Feb (32)	Mar (31)	Apr (42)	May (38)	Jun (13)	Jul (6)	Aug (11)	Sep (29)	Oct (25)	Nov (10)	Dec (11)
2016	Jan (24)	Feb (12)	Mar (13)	Apr (15)	May (22)	Jun (8)	Jul (12)	Aug (25)	Sep (8)	Oct (6)	Nov (13)	Dec (7)
2017	Jan (6)	Feb (29)	Mar (32)	Apr (8)	May (82)	Jun (42)	Jul (20)	Aug (17)	Sep (27)	Oct (14)	Nov (22)	Dec (6)
2018	Jan (12)	Feb (9)	Mar (22)	Apr (19)	May (14)	Jun (9)	Jul (9)	Aug (22)	Sep (22)	Oct (12)	Nov (13)	Dec (8)
2019	Jan (22)	Feb (3)	Mar (30)	Apr (20)	May (20)	Jun (6)	Jul (15)	Aug (25)	Sep (11)	Oct (24)	Nov (11)	Dec (6)
2020	Jan (9)	Feb (12)	Mar (29)	Apr (10)	May (22)	Jun (11)	Jul (15)	Aug (5)	Sep (6)	Oct (7)	Nov (7)	Dec (13)
2021	Jan (21)	Feb (5)	Mar (5)	Apr (6)	May (10)	Jun (7)	Jul (6)	Aug (8)	Sep (5)	Oct (9)	Nov (5)	Dec (6)
2022	Jan (5)	Feb (4)	Mar (8)	Apr (6)	May (5)	Jun (5)	Jul (10)	Aug (6)	Sep (7)	Oct (4)	Nov (4)	Dec (6)
2023	Jan (5)	Feb (5)	Mar (6)	Apr (4)	May (5)	Jun (6)	Jul (5)	Aug (5)	Sep (5)	Oct (5)	Nov (7)	Dec (8)
2024	Jan (3)	Feb (1)	Mar	Apr (2)	May	Jun (1)	Jul (1)	Aug (4)	Sep	Oct	Nov	Dec (1)
2025	Jan	Feb (2)	Mar	Apr (1)	May	Jun (1)	Jul	Aug (1)	Sep	Oct	Nov	Dec

jython-dev Mailing List for Jython (Page 51)

jython-dev — Discussion of developing and enhancing Jython itself(!) (not how to use Jython)