Menu
▾
▴
Re: [Jython-dev] Unicode user and file names (and v2.7.1)
|
From: Stefan R. <Ste...@gm...> - 2017-05-20 02:26:16
|
<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div> <div>AFAIK every release happens by having a successful RC that is renamed to 'release' after a while. So, per definition another RC is inevitable.</div> <div>That said, I suppose we should get http://bugs.jython.org/issue2487 fixed before we can release. I guess Jeff's work will be ready until then. At least that decision can be postponed until an RC is actually doable.</div> <div> <div> </div> <div> </div> <div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"> <div style="margin:0 0 10px 0;"><b>Gesendet:</b> Samstag, 20. Mai 2017 um 02:35 Uhr<br/> <b>Von:</b> "Darjus Loktevic" <da...@gm...><br/> <b>An:</b> "Jeff Allen" <ja...@fa...>, "Jython Developers" <jyt...@li...><br/> <b>Betreff:</b> Re: [Jython-dev] Unicode user and file names (and v2.7.1)</div> <div name="quoted-content"> <p>Hey Jeff,</p> <p>Sounds good. Let's do another rc but to be honest I'm not even sure the RC matters much if there aren't people trying it except us.</p> <p>Thoughts?<br/> Darjus</p> <div class="gmail_quote"> <div>On Fri, May 19, 2017, 1:19 AM Jeff Allen <<a href="mailto:ja...@fa..." onclick="parent.window.location.href='ja...@fa...'; return false;" target="_blank">ja...@fa...</a>> wrote:</div> <blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;"> <div> <p>Hi Darjus.</p> <p>On inclusion, I'm happy to go with the community view, as always. On one of the related tickets (<a class="m_1723547812866307292moz-txt-link-freetext" href="http://bugs.jython.org/issue1839" target="_blank">http://bugs.jython.org/issue1839</a>), Jim said we'd get it in if timing allowed and there was some user support.</p> <p>I'm very keen to see a 2.7.1 too. The last (soft) RC was unsuccessful, and we're still making changes, so I assume we're talking about another RC first rather than a release?</p> <p>The UTF-8 work is nearly there, but not quite: one Linux defect to fix, as noted on the same issue by James against the "latin-1" version. After all the additions in the last couple of weeks (to get full BMP support), I'm happy to find from my Linux laptop that it is still the only thing I have to do. It looks trivial. I've been unable code at all for a few days, so haven't looked into a solution, but now I'm back I expect to nail it for us today or tomorrow.</p> <p>I can, of course, merge all this myself and will. I shared your hesitancy initially, hence the fork repository, but it's turned out so well I feel it's now low risk, as long as we still have a few days.</p> <p>I will now dive under the desk and wire up my Linux dev box.</p> </div> <div> <pre class="m_1723547812866307292moz-signature">Jeff Allen</pre> </div> <div> <div class="m_1723547812866307292moz-cite-prefix">On 16/05/2017 21:46, Darjus Loktevic wrote:</div> <blockquote> <div> <div> <div> <div>Hey Jeff,<br/> </div> It seems your last commit to this branch is of three days ago. Is this ready for review? BTW, your changes look good to me.<br/> I'm a little hesitant to merge this since we've had an RC and REALLY have to release 2.7.1 It's miles better than 2.7.0.<br/> </div> Cheers,</div> Darjus</div> <div class="gmail_quote"> <div>On Mon, May 1, 2017 at 6:34 AM Jeff Allen <<a href="mailto:ja...@fa..." onclick="parent.window.location.href='ja...@fa...'; return false;" target="_blank">ja...@fa...</a>> wrote:</div> <blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;">I went for sys.getfilesystemencoding() == 'utf-8' and it works pretty<br/> well. Rather than just push directly I have published to here:<br/> <br/> <a href="https://bitbucket.org/tournesol/jython-utf8" target="_blank">https://bitbucket.org/tournesol/jython-utf8</a><br/> <br/> I write to ask for a second or third pair of eyes on it. Please tell me<br/> you can see it and whether it breaks things you care about.<br/> <br/> I touched a lot of files in the core and import system: quite a lot of<br/> tricky stuff with loaders and search paths has been adjusted. I think it<br/> a good sign that I changed hardly anything in the standard library we<br/> inherit from CPython, that we hadn't already specialised.<br/> <br/> By "works pretty well" above, I mean that the regression tests run<br/> cleanly for me when my user name is "Épreuve", where previously Jython<br/> died horribly. The launcher works from a Chinese user name too, as long<br/> as I localise Windows to China (CPython 2.7 feature). I can use the<br/> prompt and runs some tests with that setup, but I can't run the<br/> regression test yet, and printing a stack dump is fatal, so there's a<br/> bit more to do for Chinese.<br/> <br/> I think this means we have solid support for "latin-1" languages, but<br/> there are still places where we fatally assume bytes are Unicode code<br/> points.<br/> <br/> Jeff Allen<br/> <br/> On 05/04/2017 08:57, Jeff Allen wrote:<br/> > I've been working on <a href="http://bugs.jython.org/issue2356" target="_blank">http://bugs.jython.org/issue2356</a> which I'd like to<br/> > get in 2.7.1 -- it seems rather poor that Jython simply does not run for<br/> > users whose names have an un-American character ;). I know this issue is<br/> > not a blocker in most minds.<br/> ><br/> > I've made pretty good progress by allowing file names to be unicode<br/> > objects more often than they would be in CPython 2, which usually<br/> > returns them as bytes in some encoding that we may not know. I've got<br/> > the launcher to work properly, and straightened the logic in our<br/> > printing of trace-backs and exceptions from Java. Unicode file names<br/> > seems the way to go for Jython because:<br/> ><br/> > 1. Java gives us competently decoded unicode file names, from<br/> > java.io.File, etc.. Re-encoding the result will be a pain (and<br/> > overlooked).<br/> > 2. We appear not to have the codec we need ('mbcs'), that CPython<br/> > reports on Windows via sys.getfilesystemencoding().<br/> > 3. We do this already. In 2.7.0, os.getcwd() returns unicode if necessary.<br/> ><br/> > Most regression tests pass. However, I'm struggling with test_doctest.<br/> > Problems arise when mixing unicode and bytes when one byte is 128 and<br/> > over. This happens in ''.join(list) and formatted output like "%s %s" %<br/> > (ustr, bstr). The behaviour of these is identical with CPython: they<br/> > raise UnicodeDecodeError because the bytes are promoted to characters<br/> > with a strict ascii interpretation. This happens a lot in doctest.py and<br/> > traceback.py, for example, where file paths and stack dumps that include<br/> > them, are now frequently unicode, while other inputs are byte data<br/> > containing file paths presented in the console encoding.<br/> ><br/> > I can beat this into submission with enough customisation of the stdlib<br/> > modules, but that always makes me uncomfortable. I usually see that as a<br/> > hint that user code might also need to change. This may be unfounded. I<br/> > can probably ensure no impact to users of only ascii paths, and the<br/> > others seem unable to run Jython at all (in the scope of this issue).<br/> > However, I'm seriously wondering if I should pursue the approach where<br/> > file names from Java are re-encoded to bytes (maybe as utf-8<br/> > everywhere), but that's grim.<br/> ><br/> > Thoughts?<br/> ><br/> <br/> <br/> ------------------------------------------------------------------------------<br/> Check out the vibrant tech community on one of the world's most<br/> engaging tech sites, Slashdot.org! <a href="http://sdm.link/slashdot" target="_blank">http://sdm.link/slashdot</a><br/> _______________________________________________<br/> Jython-dev mailing list<br/> <a href="mailto:Jyt...@li..." onclick="parent.window.location.href='Jyt...@li...'; return false;" target="_blank">Jyt...@li...</a><br/> <a href="https://lists.sourceforge.net/lists/listinfo/jython-dev" target="_blank">https://lists.sourceforge.net/lists/listinfo/jython-dev</a></blockquote> </div> </blockquote> </div> </blockquote> </div> ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! <a href="http://sdm.link/slashdot_______________________________________________" target="_blank">http://sdm.link/slashdot_______________________________________________</a> Jython-dev mailing list Jyt...@li... <a href="https://lists.sourceforge.net/lists/listinfo/jython-dev" target="_blank">https://lists.sourceforge.net/lists/listinfo/jython-dev</a></div> </div> </div> </div></div></body></html> |
