Menu
▾
▴
Re: [Jython-dev] Unicode user and file names (and v2.7.1)
|
From: Alan K. <jyt...@xh...> - 2017-05-22 17:25:51
|
Hi folks, Great to see a solid 2.7.1 jython, and work begin in earnest on jython 3. I have only one small suggestion to make: if jython 2.7.1 is going to be one of the last 2.7 releases, maybe consider numbering it in a way that indicates it is derived from the latest version of cpython 2.7.12. This could indicate that it is as up-to-date as it can be, i.e. not derived from cpython 2.7.1 and then abandoned. Perception of abandonment is often a problem for jython: I think it's worth an effort to counter this mis-perception. Regards, Alan. On Sun, May 21, 2017 at 4:35 PM, Darjus Loktevic <da...@gm...> wrote: > Hey Guys, > > Regarding Jython3, looks like Isaiah has done a ton of work in 2016 (CCd). > Not sure how far he progressed, but indeed merging will be hard and > therefore I'd say we should not diverge further while developing on both > branches, but instead try to finalize 2.7 and switch to Jython3 full-time. > > Feel free to disagree, but here's my thinking on it: > > 1. Release Jython 2.7.1 > 2. Modernize the codebase. I think it's important for the project to > feel modern for us to attract new contributors. > 1. Java8 as the minimum (may be too much for Jython2). > 2. Github/core-workflow > 3. (Ideally) ANTLR4 for both branches, but worst case, Jython3 > only. ANTLR3 is not getting much love and ANTLR4 is quite different (does > not generate AST). > 4. Gradle, directory structure. > 3. Develop Jython3 primarily. Only bugfixes for 2.7 series. > 1. Target 3.6 (really like the typing improvements). > 2. Merge JyNI if possible. > > Cheers, > Darjus > > On Sat, May 20, 2017 at 11:45 PM Jeff Allen <ja...@fa...> wrote: > >> Thanks all. +1 on the RC. Nearly there with my bit. >> >> I have fixed the test_runpy failure James reported. It's not >> Linux-specific, just I had to quieten the unlink() error to see it on >> Windows. Bonus: we now pass the standard CPython test_runpy. The regrtest >> has been running one last time as I typed. I've pushed to >> https://bitbucket.org/tournesol/jython-utf8 just now. >> I will next merge into the Jython trunk. That may not be totally smooth >> because of the pervasive change. And now I think about it, it's worth a >> note in NEWS. My time is a little limited today, so it could be much later >> today or tomorrow evening. >> >> Jeff >> >> Jeff Allen >> >> On 20/05/2017 19:48, Jim Baker wrote: >> >> +100 >> >> On Sat, May 20, 2017 at 12:33 PM, Darjus Loktevic <da...@gm...> >> wrote: >> >>> Agreed regarding not blocking on 2487. That whole area needs a rewrite >>> and we could potentially utilize libraries available for Java 8. >>> >>> Let's get Jeff's work in and do an RC? >>> >>> Darjus >>> >>> On Fri, May 19, 2017 at 7:51 PM Jim Baker <jim...@py...> wrote: >>> >>>> I don't necessarily see http://bugs.jython.org/issue2487 as a blocker, >>>> but it would be nice. It just hasn't come up in real usage, unlike the >>>> earlier iteration of the bug which Darjus hacked around by busy waiting. I >>>> did spend some time on trying to get the publication to work without >>>> racing, using the approach I detail in that bug, but no luck yet. (But >>>> mostly because an utter lack of time to spend on the issue.) >>>> >>>> Merging in Jeff's recent work on Unicode is important and we should get >>>> it in. I haven't had a chance to test myself, but given Jeff's amazing >>>> attention to detail, I'm sure it's ready. >>>> >>>> The blocker for the RC - because we lost OSX support of setuptools >>>> support of installed executables - is fixed, as I just finally confirmed: >>>> http://bugs.jython.org/issue2570 >>>> >>>> >>>> On Fri, May 19, 2017 at 8:26 PM, Stefan Richthofer < >>>> Ste...@gm...> wrote: >>>> >>>>> AFAIK every release happens by having a successful RC that is renamed >>>>> to 'release' after a while. So, per definition another RC is inevitable. >>>>> That said, I suppose we should get http://bugs.jython.org/issue2487 >>>>> fixed before we can release. I guess Jeff's work will be ready until then. >>>>> At least that decision can be postponed until an RC is actually doable. >>>>> >>>>> >>>>> *Gesendet:* Samstag, 20. Mai 2017 um 02:35 Uhr >>>>> *Von:* "Darjus Loktevic" <da...@gm...> >>>>> *An:* "Jeff Allen" <ja...@fa...>, "Jython Developers" < >>>>> jyt...@li...> >>>>> *Betreff:* Re: [Jython-dev] Unicode user and file names (and v2.7.1) >>>>> >>>>> Hey Jeff, >>>>> >>>>> Sounds good. Let's do another rc but to be honest I'm not even sure >>>>> the RC matters much if there aren't people trying it except us. >>>>> >>>>> Thoughts? >>>>> Darjus >>>>> >>>>> On Fri, May 19, 2017, 1:19 AM Jeff Allen <ja...@fa...> wrote: >>>>> >>>>>> Hi Darjus. >>>>>> >>>>>> On inclusion, I'm happy to go with the community view, as always. On >>>>>> one of the related tickets (http://bugs.jython.org/issue1839), Jim >>>>>> said we'd get it in if timing allowed and there was some user support. >>>>>> >>>>>> I'm very keen to see a 2.7.1 too. The last (soft) RC was >>>>>> unsuccessful, and we're still making changes, so I assume we're talking >>>>>> about another RC first rather than a release? >>>>>> >>>>>> The UTF-8 work is nearly there, but not quite: one Linux defect to >>>>>> fix, as noted on the same issue by James against the "latin-1" version. >>>>>> After all the additions in the last couple of weeks (to get full BMP >>>>>> support), I'm happy to find from my Linux laptop that it is still the only >>>>>> thing I have to do. It looks trivial. I've been unable code at all for a >>>>>> few days, so haven't looked into a solution, but now I'm back I expect to >>>>>> nail it for us today or tomorrow. >>>>>> >>>>>> I can, of course, merge all this myself and will. I shared your >>>>>> hesitancy initially, hence the fork repository, but it's turned out so well >>>>>> I feel it's now low risk, as long as we still have a few days. >>>>>> >>>>>> I will now dive under the desk and wire up my Linux dev box. >>>>>> >>>>>> Jeff Allen >>>>>> >>>>>> On 16/05/2017 21:46, Darjus Loktevic wrote: >>>>>> >>>>>> Hey Jeff, >>>>>> >>>>>> It seems your last commit to this branch is of three days ago. Is >>>>>> this ready for review? BTW, your changes look good to me. >>>>>> I'm a little hesitant to merge this since we've had an RC and REALLY >>>>>> have to release 2.7.1 It's miles better than 2.7.0. >>>>>> >>>>>> Cheers, >>>>>> Darjus >>>>>> >>>>>> On Mon, May 1, 2017 at 6:34 AM Jeff Allen <ja...@fa...> wrote: >>>>>> >>>>>>> I went for sys.getfilesystemencoding() == 'utf-8' and it works pretty >>>>>>> well. Rather than just push directly I have published to here: >>>>>>> >>>>>>> https://bitbucket.org/tournesol/jython-utf8 >>>>>>> >>>>>>> I write to ask for a second or third pair of eyes on it. Please tell >>>>>>> me >>>>>>> you can see it and whether it breaks things you care about. >>>>>>> >>>>>>> I touched a lot of files in the core and import system: quite a lot >>>>>>> of >>>>>>> tricky stuff with loaders and search paths has been adjusted. I >>>>>>> think it >>>>>>> a good sign that I changed hardly anything in the standard library we >>>>>>> inherit from CPython, that we hadn't already specialised. >>>>>>> >>>>>>> By "works pretty well" above, I mean that the regression tests run >>>>>>> cleanly for me when my user name is "Épreuve", where previously >>>>>>> Jython >>>>>>> died horribly. The launcher works from a Chinese user name too, as >>>>>>> long >>>>>>> as I localise Windows to China (CPython 2.7 feature). I can use the >>>>>>> prompt and runs some tests with that setup, but I can't run the >>>>>>> regression test yet, and printing a stack dump is fatal, so there's a >>>>>>> bit more to do for Chinese. >>>>>>> >>>>>>> I think this means we have solid support for "latin-1" languages, but >>>>>>> there are still places where we fatally assume bytes are Unicode code >>>>>>> points. >>>>>>> >>>>>>> Jeff Allen >>>>>>> >>>>>>> On 05/04/2017 08:57, Jeff Allen wrote: >>>>>>> > I've been working on http://bugs.jython.org/issue2356 which I'd >>>>>>> like to >>>>>>> > get in 2.7.1 -- it seems rather poor that Jython simply does not >>>>>>> run for >>>>>>> > users whose names have an un-American character ;). I know this >>>>>>> issue is >>>>>>> > not a blocker in most minds. >>>>>>> > >>>>>>> > I've made pretty good progress by allowing file names to be unicode >>>>>>> > objects more often than they would be in CPython 2, which usually >>>>>>> > returns them as bytes in some encoding that we may not know. I've >>>>>>> got >>>>>>> > the launcher to work properly, and straightened the logic in our >>>>>>> > printing of trace-backs and exceptions from Java. Unicode file >>>>>>> names >>>>>>> > seems the way to go for Jython because: >>>>>>> > >>>>>>> > 1. Java gives us competently decoded unicode file names, from >>>>>>> > java.io.File, etc.. Re-encoding the result will be a pain (and >>>>>>> > overlooked). >>>>>>> > 2. We appear not to have the codec we need ('mbcs'), that CPython >>>>>>> > reports on Windows via sys.getfilesystemencoding(). >>>>>>> > 3. We do this already. In 2.7.0, os.getcwd() returns unicode if >>>>>>> necessary. >>>>>>> > >>>>>>> > Most regression tests pass. However, I'm struggling with >>>>>>> test_doctest. >>>>>>> > Problems arise when mixing unicode and bytes when one byte is 128 >>>>>>> and >>>>>>> > over. This happens in ''.join(list) and formatted output like "%s >>>>>>> %s" % >>>>>>> > (ustr, bstr). The behaviour of these is identical with CPython: >>>>>>> they >>>>>>> > raise UnicodeDecodeError because the bytes are promoted to >>>>>>> characters >>>>>>> > with a strict ascii interpretation. This happens a lot in >>>>>>> doctest.py and >>>>>>> > traceback.py, for example, where file paths and stack dumps that >>>>>>> include >>>>>>> > them, are now frequently unicode, while other inputs are byte data >>>>>>> > containing file paths presented in the console encoding. >>>>>>> > >>>>>>> > I can beat this into submission with enough customisation of the >>>>>>> stdlib >>>>>>> > modules, but that always makes me uncomfortable. I usually see >>>>>>> that as a >>>>>>> > hint that user code might also need to change. This may be >>>>>>> unfounded. I >>>>>>> > can probably ensure no impact to users of only ascii paths, and the >>>>>>> > others seem unable to run Jython at all (in the scope of this >>>>>>> issue). >>>>>>> > However, I'm seriously wondering if I should pursue the approach >>>>>>> where >>>>>>> > file names from Java are re-encoded to bytes (maybe as utf-8 >>>>>>> > everywhere), but that's grim. >>>>>>> > >>>>>>> > Thoughts? >>>>>>> > >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------ >>>>>>> ------------------ >>>>>>> Check out the vibrant tech community on one of the world's most >>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>>>>> _______________________________________________ >>>>>>> Jython-dev mailing list >>>>>>> Jyt...@li... >>>>>>> https://lists.sourceforge.net/lists/listinfo/jython-dev >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>> Check out the vibrant tech community on one of the world's most engaging >>>>> tech sites, Slashdot.org! http://sdm.link/slashdot______ >>>>> _________________________________________ Jython-dev mailing list >>>>> Jyt...@li... https://lists.sourceforge.net/ >>>>> lists/listinfo/jython-dev >>>>> >>>>> ------------------------------------------------------------ >>>>> ------------------ >>>>> Check out the vibrant tech community on one of the world's most >>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>>> _______________________________________________ >>>>> Jython-dev mailing list >>>>> Jyt...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/jython-dev >>>>> >>>>> >>>> >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> >> >> >> _______________________________________________ >> Jython-dev mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/jython-dev >> >> >> > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev > > |
