Menu
▾
▴
Re: [Jython-dev] Unicode user and file names (and v2.7.1)
|
From: Darjus L. <da...@gm...> - 2017-05-20 18:33:40
|
Agreed regarding not blocking on 2487. That whole area needs a rewrite and we could potentially utilize libraries available for Java 8. Let's get Jeff's work in and do an RC? Darjus On Fri, May 19, 2017 at 7:51 PM Jim Baker <jim...@py...> wrote: > I don't necessarily see http://bugs.jython.org/issue2487 as a blocker, > but it would be nice. It just hasn't come up in real usage, unlike the > earlier iteration of the bug which Darjus hacked around by busy waiting. I > did spend some time on trying to get the publication to work without > racing, using the approach I detail in that bug, but no luck yet. (But > mostly because an utter lack of time to spend on the issue.) > > Merging in Jeff's recent work on Unicode is important and we should get it > in. I haven't had a chance to test myself, but given Jeff's amazing > attention to detail, I'm sure it's ready. > > The blocker for the RC - because we lost OSX support of setuptools support > of installed executables - is fixed, as I just finally confirmed: > http://bugs.jython.org/issue2570 > > > On Fri, May 19, 2017 at 8:26 PM, Stefan Richthofer < > Ste...@gm...> wrote: > >> AFAIK every release happens by having a successful RC that is renamed to >> 'release' after a while. So, per definition another RC is inevitable. >> That said, I suppose we should get http://bugs.jython.org/issue2487 >> fixed before we can release. I guess Jeff's work will be ready until then. >> At least that decision can be postponed until an RC is actually doable. >> >> >> *Gesendet:* Samstag, 20. Mai 2017 um 02:35 Uhr >> *Von:* "Darjus Loktevic" <da...@gm...> >> *An:* "Jeff Allen" <ja...@fa...>, "Jython Developers" < >> jyt...@li...> >> *Betreff:* Re: [Jython-dev] Unicode user and file names (and v2.7.1) >> >> Hey Jeff, >> >> Sounds good. Let's do another rc but to be honest I'm not even sure the >> RC matters much if there aren't people trying it except us. >> >> Thoughts? >> Darjus >> >> On Fri, May 19, 2017, 1:19 AM Jeff Allen <ja...@fa...> wrote: >> >>> Hi Darjus. >>> >>> On inclusion, I'm happy to go with the community view, as always. On one >>> of the related tickets (http://bugs.jython.org/issue1839), Jim said >>> we'd get it in if timing allowed and there was some user support. >>> >>> I'm very keen to see a 2.7.1 too. The last (soft) RC was unsuccessful, >>> and we're still making changes, so I assume we're talking about another RC >>> first rather than a release? >>> >>> The UTF-8 work is nearly there, but not quite: one Linux defect to fix, >>> as noted on the same issue by James against the "latin-1" version. After >>> all the additions in the last couple of weeks (to get full BMP support), >>> I'm happy to find from my Linux laptop that it is still the only thing I >>> have to do. It looks trivial. I've been unable code at all for a few days, >>> so haven't looked into a solution, but now I'm back I expect to nail it for >>> us today or tomorrow. >>> >>> I can, of course, merge all this myself and will. I shared your >>> hesitancy initially, hence the fork repository, but it's turned out so well >>> I feel it's now low risk, as long as we still have a few days. >>> >>> I will now dive under the desk and wire up my Linux dev box. >>> >>> Jeff Allen >>> >>> On 16/05/2017 21:46, Darjus Loktevic wrote: >>> >>> Hey Jeff, >>> >>> It seems your last commit to this branch is of three days ago. Is this >>> ready for review? BTW, your changes look good to me. >>> I'm a little hesitant to merge this since we've had an RC and REALLY >>> have to release 2.7.1 It's miles better than 2.7.0. >>> >>> Cheers, >>> Darjus >>> >>> On Mon, May 1, 2017 at 6:34 AM Jeff Allen <ja...@fa...> wrote: >>> >>>> I went for sys.getfilesystemencoding() == 'utf-8' and it works pretty >>>> well. Rather than just push directly I have published to here: >>>> >>>> https://bitbucket.org/tournesol/jython-utf8 >>>> >>>> I write to ask for a second or third pair of eyes on it. Please tell me >>>> you can see it and whether it breaks things you care about. >>>> >>>> I touched a lot of files in the core and import system: quite a lot of >>>> tricky stuff with loaders and search paths has been adjusted. I think it >>>> a good sign that I changed hardly anything in the standard library we >>>> inherit from CPython, that we hadn't already specialised. >>>> >>>> By "works pretty well" above, I mean that the regression tests run >>>> cleanly for me when my user name is "Épreuve", where previously Jython >>>> died horribly. The launcher works from a Chinese user name too, as long >>>> as I localise Windows to China (CPython 2.7 feature). I can use the >>>> prompt and runs some tests with that setup, but I can't run the >>>> regression test yet, and printing a stack dump is fatal, so there's a >>>> bit more to do for Chinese. >>>> >>>> I think this means we have solid support for "latin-1" languages, but >>>> there are still places where we fatally assume bytes are Unicode code >>>> points. >>>> >>>> Jeff Allen >>>> >>>> On 05/04/2017 08:57, Jeff Allen wrote: >>>> > I've been working on http://bugs.jython.org/issue2356 which I'd like >>>> to >>>> > get in 2.7.1 -- it seems rather poor that Jython simply does not run >>>> for >>>> > users whose names have an un-American character ;). I know this issue >>>> is >>>> > not a blocker in most minds. >>>> > >>>> > I've made pretty good progress by allowing file names to be unicode >>>> > objects more often than they would be in CPython 2, which usually >>>> > returns them as bytes in some encoding that we may not know. I've got >>>> > the launcher to work properly, and straightened the logic in our >>>> > printing of trace-backs and exceptions from Java. Unicode file names >>>> > seems the way to go for Jython because: >>>> > >>>> > 1. Java gives us competently decoded unicode file names, from >>>> > java.io.File, etc.. Re-encoding the result will be a pain (and >>>> > overlooked). >>>> > 2. We appear not to have the codec we need ('mbcs'), that CPython >>>> > reports on Windows via sys.getfilesystemencoding(). >>>> > 3. We do this already. In 2.7.0, os.getcwd() returns unicode if >>>> necessary. >>>> > >>>> > Most regression tests pass. However, I'm struggling with test_doctest. >>>> > Problems arise when mixing unicode and bytes when one byte is 128 and >>>> > over. This happens in ''.join(list) and formatted output like "%s %s" >>>> % >>>> > (ustr, bstr). The behaviour of these is identical with CPython: they >>>> > raise UnicodeDecodeError because the bytes are promoted to characters >>>> > with a strict ascii interpretation. This happens a lot in doctest.py >>>> and >>>> > traceback.py, for example, where file paths and stack dumps that >>>> include >>>> > them, are now frequently unicode, while other inputs are byte data >>>> > containing file paths presented in the console encoding. >>>> > >>>> > I can beat this into submission with enough customisation of the >>>> stdlib >>>> > modules, but that always makes me uncomfortable. I usually see that >>>> as a >>>> > hint that user code might also need to change. This may be unfounded. >>>> I >>>> > can probably ensure no impact to users of only ascii paths, and the >>>> > others seem unable to run Jython at all (in the scope of this issue). >>>> > However, I'm seriously wondering if I should pursue the approach where >>>> > file names from Java are re-encoded to bytes (maybe as utf-8 >>>> > everywhere), but that's grim. >>>> > >>>> > Thoughts? >>>> > >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Check out the vibrant tech community on one of the world's most >>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>> _______________________________________________ >>>> Jython-dev mailing list >>>> Jyt...@li... >>>> https://lists.sourceforge.net/lists/listinfo/jython-dev >>> >>> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most engaging >> tech sites, Slashdot.org! >> http://sdm.link/slashdot_______________________________________________ >> Jython-dev mailing list Jyt...@li... >> https://lists.sourceforge.net/lists/listinfo/jython-dev >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> _______________________________________________ >> Jython-dev mailing list >> Jyt...@li... >> https://lists.sourceforge.net/lists/listinfo/jython-dev >> >> > |
