Menu
▾
▴
Re: [Jython-dev] Unicode user and file names (and v2.7.1)
|
From: Stefan R. <Ste...@gm...> - 2017-05-21 00:01:11
|
<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div> <div>I'd like to have http://bugs.jython.org/issue2536 "fixed" by removing that infinite recursion test; maybe we could even close this annoying issue then. A new Jython release should have reliable test suite. However I don't know the best way to remove or blacklist a certain test.</div> <div>Jeff, that's your domain IIRC...? Suggestions?</div> <div> </div> <div>Also, we might want to update (some) extlibs to current versions, e.g. I tested current Guava while working on #2536 two months ago and found no issues with it (at least not regarding regrtests). http://bugs.jython.org/issue2582 requires a current jnr version in order to decide further steps on that issue. There are probably more. I don't suggest to spend much time on this. Just do updates that cause no issues and leave everything else as it is. Yes the list of extlibs to check is long. Maybe we can split this work between us and maybe more volunteers?</div> <div> </div> <div>@Jim: Could you share some more insights about #2487, what you tried and why it failed. I'd like to give it at least anther look before next RC.</div> <div> </div> <div>I once tried to update lib-python to CPython 2.7.13 version, but that caused lots of additional regrtest failures, 35 or so. Would probably take another year to resolve this stuff. I guess that time would be better invested in Jython 3. However, maybe we can revisit this for Jython 2.7.2.</div> <div> </div> <div>BTW, has anyone thought about a path for porting Jython 2.7.1 features and fixes to Jython 3? Jython 3 was forked shortly after 2.7.0 release and already received some serious amount of work. So we already have a notable divergence between Jython 3 and Jython 2.7.1. IMO this is a high priority topic right after Jython 2.7.1 release, because every further piece of 2.7.x work can make this worse.</div> <div> </div> <div> </div> <div>-Stefan</div> <div> </div> <div> <div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"> <div style="margin:0 0 10px 0;"><b>Gesendet:</b> Samstag, 20. Mai 2017 um 20:48 Uhr<br/> <b>Von:</b> "Jim Baker" <jim...@py...><br/> <b>An:</b> "Darjus Loktevic" <da...@gm...><br/> <b>Cc:</b> "Stefan Richthofer" <Ste...@gm...>, "Jython Developers" <jyt...@li...><br/> <b>Betreff:</b> Re: [Jython-dev] Unicode user and file names (and v2.7.1)</div> <div name="quoted-content"> <div>+100</div> <div class="gmail_extra"> <div class="gmail_quote">On Sat, May 20, 2017 at 12:33 PM, Darjus Loktevic <span><<a href="mailto:da...@gm..." onclick="parent.window.location.href='da...@gm...'; return false;" target="_blank">da...@gm...</a>></span> wrote: <blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;"> <div> <div> <div>Agreed regarding not blocking on 2487. That whole area needs a rewrite and we could potentially utilize libraries available for Java 8.<br/> </div> Let's get Jeff's work in and do an RC?<br/> </div> <span class="HOEnZb"><font color="#888888">Darjus</font></span></div> <div class="HOEnZb"> <div class="h5"> <div class="gmail_quote"> <div>On Fri, May 19, 2017 at 7:51 PM Jim Baker <<a href="mailto:jim...@py..." onclick="parent.window.location.href='jim...@py...'; return false;" target="_blank">jim...@py...</a>> wrote:</div> <blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;"> <div>I don't necessarily see <a href="http://bugs.jython.org/issue2487" target="_blank">http://bugs.jython.org/issue2487</a> as a blocker, but it would be nice. It just hasn't come up in real usage, unlike the earlier iteration of the bug which Darjus hacked around by busy waiting. I did spend some time on trying to get the publication to work without racing, using the approach I detail in that bug, but no luck yet. (But mostly because an utter lack of time to spend on the issue.) <div> </div> <div>Merging in Jeff's recent work on Unicode is important and we should get it in. I haven't had a chance to test myself, but given Jeff's amazing attention to detail, I'm sure it's ready.</div> <div> </div> <div>The blocker for the RC - because we lost OSX support of setuptools support of installed executables - is fixed, as I just finally confirmed: <a href="http://bugs.jython.org/issue2570" target="_blank">http://bugs.jython.org/issue2570</a></div> <div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote">On Fri, May 19, 2017 at 8:26 PM, Stefan Richthofer <span><<a href="mailto:Ste...@gm..." onclick="parent.window.location.href='Ste...@gm...'; return false;" target="_blank">Ste...@gm...</a>></span> wrote: <blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;"> <div> <div style="font-family: Verdana;font-size: 12.0px;"> <div> <div>AFAIK every release happens by having a successful RC that is renamed to 'release' after a while. So, per definition another RC is inevitable.</div> <div>That said, I suppose we should get <a href="http://bugs.jython.org/issue2487" target="_blank">http://bugs.jython.org/issue2487</a> fixed before we can release. I guess Jeff's work will be ready until then. At least that decision can be postponed until an RC is actually doable.</div> <div> <div> </div> <div> </div> <div style="margin: 10.0px 5.0px 5.0px 10.0px;padding: 10.0px 0 10.0px 10.0px;border-left: 2.0px solid rgb(195,217,229);"> <div style="margin: 0 0 10.0px 0;"><b>Gesendet:</b> Samstag, 20. Mai 2017 um 02:35 Uhr<br/> <b>Von:</b> "Darjus Loktevic" <<a href="mailto:da...@gm..." onclick="parent.window.location.href='da...@gm...'; return false;" target="_blank">da...@gm...</a>><br/> <b>An:</b> "Jeff Allen" <<a href="mailto:ja...@fa..." onclick="parent.window.location.href='ja...@fa...'; return false;" target="_blank">ja...@fa...</a>>, "Jython Developers" <<a href="mailto:jyt...@li..." onclick="parent.window.location.href='jyt...@li...'; return false;" target="_blank">jyt...@li...</a>><br/> <b>Betreff:</b> Re: [Jython-dev] Unicode user and file names (and v2.7.1)</div> <div> <div> <div class="m_8810925058365348314m_-8779070674509468614h5"> <p>Hey Jeff,</p> <p>Sounds good. Let's do another rc but to be honest I'm not even sure the RC matters much if there aren't people trying it except us.</p> <p>Thoughts?<br/> Darjus</p> <div class="gmail_quote"> <div>On Fri, May 19, 2017, 1:19 AM Jeff Allen <<a href="mailto:ja...@fa..." onclick="parent.window.location.href='ja...@fa...'; return false;" target="_blank">ja...@fa...</a>> wrote:</div> <blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;"> <div> <p>Hi Darjus.</p> <p>On inclusion, I'm happy to go with the community view, as always. On one of the related tickets (<a class="m_8810925058365348314m_-8779070674509468614m_-528727688133917487m_1723547812866307292moz-txt-link-freetext" href="http://bugs.jython.org/issue1839" target="_blank">http://bugs.jython.org/issue1839</a>), Jim said we'd get it in if timing allowed and there was some user support.</p> <p>I'm very keen to see a 2.7.1 too. The last (soft) RC was unsuccessful, and we're still making changes, so I assume we're talking about another RC first rather than a release?</p> <p>The UTF-8 work is nearly there, but not quite: one Linux defect to fix, as noted on the same issue by James against the "latin-1" version. After all the additions in the last couple of weeks (to get full BMP support), I'm happy to find from my Linux laptop that it is still the only thing I have to do. It looks trivial. I've been unable code at all for a few days, so haven't looked into a solution, but now I'm back I expect to nail it for us today or tomorrow.</p> <p>I can, of course, merge all this myself and will. I shared your hesitancy initially, hence the fork repository, but it's turned out so well I feel it's now low risk, as long as we still have a few days.</p> <p>I will now dive under the desk and wire up my Linux dev box.</p> </div> <div> <pre class="m_8810925058365348314m_-8779070674509468614m_-528727688133917487m_1723547812866307292moz-signature">Jeff Allen</pre> </div> <div> <div class="m_8810925058365348314m_-8779070674509468614m_-528727688133917487m_1723547812866307292moz-cite-prefix">On 16/05/2017 21:46, Darjus Loktevic wrote:</div> <blockquote> <div> <div> <div> <div>Hey Jeff,<br/> </div> It seems your last commit to this branch is of three days ago. Is this ready for review? BTW, your changes look good to me.<br/> I'm a little hesitant to merge this since we've had an RC and REALLY have to release 2.7.1 It's miles better than 2.7.0.<br/> </div> Cheers,</div> Darjus</div> <div class="gmail_quote"> <div>On Mon, May 1, 2017 at 6:34 AM Jeff Allen <<a href="mailto:ja...@fa..." onclick="parent.window.location.href='ja...@fa...'; return false;" target="_blank">ja...@fa...</a>> wrote:</div> <blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;">I went for sys.getfilesystemencoding() == 'utf-8' and it works pretty<br/> well. Rather than just push directly I have published to here:<br/> <br/> <a href="https://bitbucket.org/tournesol/jython-utf8" target="_blank">https://bitbucket.org/tournesol/jython-utf8</a><br/> <br/> I write to ask for a second or third pair of eyes on it. Please tell me<br/> you can see it and whether it breaks things you care about.<br/> <br/> I touched a lot of files in the core and import system: quite a lot of<br/> tricky stuff with loaders and search paths has been adjusted. I think it<br/> a good sign that I changed hardly anything in the standard library we<br/> inherit from CPython, that we hadn't already specialised.<br/> <br/> By "works pretty well" above, I mean that the regression tests run<br/> cleanly for me when my user name is "Épreuve", where previously Jython<br/> died horribly. The launcher works from a Chinese user name too, as long<br/> as I localise Windows to China (CPython 2.7 feature). I can use the<br/> prompt and runs some tests with that setup, but I can't run the<br/> regression test yet, and printing a stack dump is fatal, so there's a<br/> bit more to do for Chinese.<br/> <br/> I think this means we have solid support for "latin-1" languages, but<br/> there are still places where we fatally assume bytes are Unicode code<br/> points.<br/> <br/> Jeff Allen<br/> <br/> On 05/04/2017 08:57, Jeff Allen wrote:<br/> > I've been working on <a href="http://bugs.jython.org/issue2356" target="_blank">http://bugs.jython.org/issue2356</a> which I'd like to<br/> > get in 2.7.1 -- it seems rather poor that Jython simply does not run for<br/> > users whose names have an un-American character ;). I know this issue is<br/> > not a blocker in most minds.<br/> ><br/> > I've made pretty good progress by allowing file names to be unicode<br/> > objects more often than they would be in CPython 2, which usually<br/> > returns them as bytes in some encoding that we may not know. I've got<br/> > the launcher to work properly, and straightened the logic in our<br/> > printing of trace-backs and exceptions from Java. Unicode file names<br/> > seems the way to go for Jython because:<br/> ><br/> > 1. Java gives us competently decoded unicode file names, from<br/> > java.io.File, etc.. Re-encoding the result will be a pain (and<br/> > overlooked).<br/> > 2. We appear not to have the codec we need ('mbcs'), that CPython<br/> > reports on Windows via sys.getfilesystemencoding().<br/> > 3. We do this already. In 2.7.0, os.getcwd() returns unicode if necessary.<br/> ><br/> > Most regression tests pass. However, I'm struggling with test_doctest.<br/> > Problems arise when mixing unicode and bytes when one byte is 128 and<br/> > over. This happens in ''.join(list) and formatted output like "%s %s" %<br/> > (ustr, bstr). The behaviour of these is identical with CPython: they<br/> > raise UnicodeDecodeError because the bytes are promoted to characters<br/> > with a strict ascii interpretation. This happens a lot in doctest.py and<br/> > traceback.py, for example, where file paths and stack dumps that include<br/> > them, are now frequently unicode, while other inputs are byte data<br/> > containing file paths presented in the console encoding.<br/> ><br/> > I can beat this into submission with enough customisation of the stdlib<br/> > modules, but that always makes me uncomfortable. I usually see that as a<br/> > hint that user code might also need to change. This may be unfounded. I<br/> > can probably ensure no impact to users of only ascii paths, and the<br/> > others seem unable to run Jython at all (in the scope of this issue).<br/> > However, I'm seriously wondering if I should pursue the approach where<br/> > file names from Java are re-encoded to bytes (maybe as utf-8<br/> > everywhere), but that's grim.<br/> ><br/> > Thoughts?<br/> ><br/> <br/> <br/> ------------------------------------------------------------------------------<br/> Check out the vibrant tech community on one of the world's most<br/> engaging tech sites, Slashdot.org! <a href="http://sdm.link/slashdot" target="_blank">http://sdm.link/slashdot</a><br/> _______________________________________________<br/> Jython-dev mailing list<br/> <a href="mailto:Jyt...@li..." onclick="parent.window.location.href='Jyt...@li...'; return false;" target="_blank">Jyt...@li...</a><br/> <a href="https://lists.sourceforge.net/lists/listinfo/jython-dev" target="_blank">https://lists.sourceforge.net/lists/listinfo/jython-dev</a></blockquote> </div> </blockquote> </div> </blockquote> </div> </div> </div> ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! <a href="http://sdm.link/slashdot_______________________________________________" target="_blank">http://sdm.link/slashdot_______________________________________________</a> Jython-dev mailing list <a href="mailto:Jyt...@li..." onclick="parent.window.location.href='Jyt...@li...'; return false;" target="_blank">Jyt...@li...</a> <a href="https://lists.sourceforge.net/lists/listinfo/jython-dev" target="_blank">https://lists.sourceforge.net/lists/listinfo/jython-dev</a></div> </div> </div> </div> </div> </div> <br/> ------------------------------------------------------------------------------<br/> Check out the vibrant tech community on one of the world's most<br/> engaging tech sites, Slashdot.org! <a href="http://sdm.link/slashdot" target="_blank">http://sdm.link/slashdot</a><br/> _______________________________________________<br/> Jython-dev mailing list<br/> <a href="mailto:Jyt...@li..." onclick="parent.window.location.href='Jyt...@li...'; return false;" target="_blank">Jyt...@li...</a><br/> <a href="https://lists.sourceforge.net/lists/listinfo/jython-dev" target="_blank">https://lists.sourceforge.net/lists/listinfo/jython-dev</a><br/> </blockquote> </div> </div> </blockquote> </div> </div> </div> </blockquote> </div> </div> </div> </div> </div> </div></div></body></html> |
