Re: [Jython-dev] Unicode user and file names (and v2.7.1)

[Jim B. <jim...@py...>]

Alan,

That's a great suggestion. 2.7 was specifically chosen to show this
correspondence. In the past, we were not so as focused on compatibility,
but most of the changes — and corresponding delays — in what we have been
planning to call 2.7.1 are because of the continued development on CPython
2.7, by backporting fixes from successive versions of CPython 3.

So calling it 2.7.12 helps illustrate this. Any other thoughts on Alan's
proposal?

- Jim

On Mon, May 22, 2017 at 11:25 AM, Alan Kennedy <jyt...@xh...> wrote:

> Hi folks,
>
> Great to see a solid 2.7.1 jython, and work begin in earnest on jython 3.
>
> I have only one small suggestion to make: if jython 2.7.1 is going to be
> one of the last 2.7 releases, maybe consider numbering it in a way that
> indicates it is derived from the latest version of cpython 2.7.12. This
> could indicate that it is as up-to-date as it can be, i.e. not derived from
> cpython 2.7.1 and then abandoned.
>
> Perception of abandonment is often a problem for jython: I think it's
> worth an effort to counter this mis-perception.
>
> Regards,
>
> Alan.
>
>
> On Sun, May 21, 2017 at 4:35 PM, Darjus Loktevic <da...@gm...> wrote:
>
>> Hey Guys,
>>
>> Regarding Jython3, looks like Isaiah has done a ton of work in 2016
>> (CCd). Not sure how far he progressed, but indeed merging will be hard and
>> therefore I'd say we should not diverge further while developing on both
>> branches, but instead try to finalize 2.7 and switch to Jython3 full-time.
>>
>> Feel free to disagree, but here's my thinking on it:
>>
>>    1. Release Jython 2.7.1
>>    2. Modernize the codebase. I think it's important for the project to
>>    feel modern for us to attract new contributors.
>>    1. Java8 as the minimum (may be too much for Jython2).
>>       2. Github/core-workflow
>>       3. (Ideally) ANTLR4 for both branches, but worst case, Jython3
>>       only. ANTLR3 is not getting much love and ANTLR4 is quite different (does
>>       not generate AST).
>>       4. Gradle, directory structure.
>>    3. Develop Jython3 primarily. Only bugfixes for 2.7 series.
>>       1. Target 3.6 (really like the typing improvements).
>>       2. Merge JyNI if possible.
>>
>> Cheers,
>> Darjus
>>
>> On Sat, May 20, 2017 at 11:45 PM Jeff Allen <ja...@fa...> wrote:
>>
>>> Thanks all. +1 on the RC. Nearly there with my bit.
>>>
>>> I have fixed the  test_runpy failure James reported. It's not
>>> Linux-specific, just I had to quieten the unlink() error to see it on
>>> Windows. Bonus: we now pass the standard CPython test_runpy. The regrtest
>>> has been running one last time as I typed. I've pushed to
>>> https://bitbucket.org/tournesol/jython-utf8 just now.
>>> I will next merge into the Jython trunk. That may not be totally smooth
>>> because of the pervasive change. And now I think about it, it's worth a
>>> note in NEWS. My time is a little limited today, so it could be much later
>>> today or tomorrow evening.
>>>
>>> Jeff
>>>
>>> Jeff Allen
>>>
>>> On 20/05/2017 19:48, Jim Baker wrote:
>>>
>>> +100
>>>
>>> On Sat, May 20, 2017 at 12:33 PM, Darjus Loktevic <da...@gm...>
>>> wrote:
>>>
>>>> Agreed regarding not blocking on 2487. That whole area needs a rewrite
>>>> and we could potentially utilize libraries available for Java 8.
>>>>
>>>> Let's get Jeff's work in and do an RC?
>>>>
>>>> Darjus
>>>>
>>>> On Fri, May 19, 2017 at 7:51 PM Jim Baker <jim...@py...> wrote:
>>>>
>>>>> I don't necessarily see http://bugs.jython.org/issue2487 as a
>>>>> blocker, but it would be nice. It just hasn't come up in real usage, unlike
>>>>> the earlier iteration of the bug which Darjus hacked around by busy
>>>>> waiting. I did spend some time on trying to get the publication to work
>>>>> without racing, using the approach I detail in that bug, but no luck yet.
>>>>> (But mostly because an utter lack of time to spend on the issue.)
>>>>>
>>>>> Merging in Jeff's recent work on Unicode is important and we should
>>>>> get it in. I haven't had a chance to test myself, but given Jeff's amazing
>>>>> attention to detail, I'm sure it's ready.
>>>>>
>>>>> The blocker for the RC - because we lost OSX support of setuptools
>>>>> support of installed executables - is fixed, as I just finally confirmed:
>>>>> http://bugs.jython.org/issue2570
>>>>>
>>>>>
>>>>> On Fri, May 19, 2017 at 8:26 PM, Stefan Richthofer <
>>>>> Ste...@gm...> wrote:
>>>>>
>>>>>> AFAIK every release happens by having a successful RC that is renamed
>>>>>> to 'release' after a while. So, per definition another RC is inevitable.
>>>>>> That said, I suppose we should get http://bugs.jython.org/issue2487
>>>>>> fixed before we can release. I guess Jeff's work will be ready until then.
>>>>>> At least that decision can be postponed until an RC is actually doable.
>>>>>>
>>>>>>
>>>>>> *Gesendet:* Samstag, 20. Mai 2017 um 02:35 Uhr
>>>>>> *Von:* "Darjus Loktevic" <da...@gm...>
>>>>>> *An:* "Jeff Allen" <ja...@fa...>, "Jython Developers" <
>>>>>> jyt...@li...>
>>>>>> *Betreff:* Re: [Jython-dev] Unicode user and file names (and v2.7.1)
>>>>>>
>>>>>> Hey Jeff,
>>>>>>
>>>>>> Sounds good. Let's do another rc but to be honest I'm not even sure
>>>>>> the RC matters much if there aren't people trying it except us.
>>>>>>
>>>>>> Thoughts?
>>>>>> Darjus
>>>>>>
>>>>>> On Fri, May 19, 2017, 1:19 AM Jeff Allen <ja...@fa...> wrote:
>>>>>>
>>>>>>> Hi Darjus.
>>>>>>>
>>>>>>> On inclusion, I'm happy to go with the community view, as always. On
>>>>>>> one of the related tickets (http://bugs.jython.org/issue1839), Jim
>>>>>>> said we'd get it in if timing allowed and there was some user support.
>>>>>>>
>>>>>>> I'm very keen to see a 2.7.1 too. The last (soft) RC was
>>>>>>> unsuccessful, and we're still making changes, so I assume we're talking
>>>>>>> about another RC first rather than a release?
>>>>>>>
>>>>>>> The UTF-8 work is nearly there, but not quite: one Linux defect to
>>>>>>> fix, as noted on the same issue by James against the "latin-1" version.
>>>>>>> After all the additions in the last couple of weeks (to get full BMP
>>>>>>> support), I'm happy to find from my Linux laptop that it is still the only
>>>>>>> thing I have to do. It looks trivial. I've been unable code at all for a
>>>>>>> few days, so haven't looked into a solution, but now I'm back I expect to
>>>>>>> nail it for us today or tomorrow.
>>>>>>>
>>>>>>> I can, of course, merge all this myself and will. I shared your
>>>>>>> hesitancy initially, hence the fork repository, but it's turned out so well
>>>>>>> I feel it's now low risk, as long as we still have a few days.
>>>>>>>
>>>>>>> I will now dive under the desk and wire up my Linux dev box.
>>>>>>>
>>>>>>> Jeff Allen
>>>>>>>
>>>>>>> On 16/05/2017 21:46, Darjus Loktevic wrote:
>>>>>>>
>>>>>>> Hey Jeff,
>>>>>>>
>>>>>>> It seems your last commit to this branch is of three days ago. Is
>>>>>>> this ready for review? BTW, your changes look good to me.
>>>>>>> I'm a little hesitant to merge this since we've had an RC and REALLY
>>>>>>> have to release 2.7.1 It's miles better than 2.7.0.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Darjus
>>>>>>>
>>>>>>> On Mon, May 1, 2017 at 6:34 AM Jeff Allen <ja...@fa...>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I went for sys.getfilesystemencoding() == 'utf-8' and it works
>>>>>>>> pretty
>>>>>>>> well. Rather than just push directly I have published to here:
>>>>>>>>
>>>>>>>> https://bitbucket.org/tournesol/jython-utf8
>>>>>>>>
>>>>>>>> I write to ask for a second or third pair of eyes on it. Please
>>>>>>>> tell me
>>>>>>>> you can see it and whether it breaks things you care about.
>>>>>>>>
>>>>>>>> I touched a lot of files in the core and import system: quite a lot
>>>>>>>> of
>>>>>>>> tricky stuff with loaders and search paths has been adjusted. I
>>>>>>>> think it
>>>>>>>> a good sign that I changed hardly anything in the standard library
>>>>>>>> we
>>>>>>>> inherit from CPython, that we hadn't already specialised.
>>>>>>>>
>>>>>>>> By "works pretty well" above, I mean that the regression tests run
>>>>>>>> cleanly for me when my user name is "Épreuve", where previously
>>>>>>>> Jython
>>>>>>>> died horribly. The launcher works from a Chinese user name too, as
>>>>>>>> long
>>>>>>>> as I localise Windows to China (CPython 2.7 feature). I can use the
>>>>>>>> prompt and runs some tests with that setup, but I can't run the
>>>>>>>> regression test yet, and printing a stack dump is fatal, so there's
>>>>>>>> a
>>>>>>>> bit more to do for Chinese.
>>>>>>>>
>>>>>>>> I think this means we have solid support for "latin-1" languages,
>>>>>>>> but
>>>>>>>> there are still places where we fatally assume bytes are Unicode
>>>>>>>> code
>>>>>>>> points.
>>>>>>>>
>>>>>>>> Jeff Allen
>>>>>>>>
>>>>>>>> On 05/04/2017 08:57, Jeff Allen wrote:
>>>>>>>> > I've been working on http://bugs.jython.org/issue2356 which I'd
>>>>>>>> like to
>>>>>>>> > get in 2.7.1 -- it seems rather poor that Jython simply does not
>>>>>>>> run for
>>>>>>>> > users whose names have an un-American character ;). I know this
>>>>>>>> issue is
>>>>>>>> > not a blocker in most minds.
>>>>>>>> >
>>>>>>>> > I've made pretty good progress by allowing file names to be
>>>>>>>> unicode
>>>>>>>> > objects more often than they would be in CPython 2, which usually
>>>>>>>> > returns them as bytes in some encoding that we may not know. I've
>>>>>>>> got
>>>>>>>> > the launcher to work properly, and straightened the logic in our
>>>>>>>> > printing of trace-backs and exceptions from Java. Unicode file
>>>>>>>> names
>>>>>>>> > seems the way to go for Jython because:
>>>>>>>> >
>>>>>>>> >   1. Java gives us competently decoded unicode file names, from
>>>>>>>> >      java.io.File, etc.. Re-encoding the result will be a pain
>>>>>>>> (and
>>>>>>>> >      overlooked).
>>>>>>>> >   2. We appear not to have the codec we need ('mbcs'), that
>>>>>>>> CPython
>>>>>>>> >      reports on Windows via sys.getfilesystemencoding().
>>>>>>>> >   3. We do this already. In 2.7.0, os.getcwd() returns unicode if
>>>>>>>> necessary.
>>>>>>>> >
>>>>>>>> > Most regression tests pass. However, I'm struggling with
>>>>>>>> test_doctest.
>>>>>>>> > Problems arise when mixing unicode and bytes when one byte is 128
>>>>>>>> and
>>>>>>>> > over. This happens in ''.join(list) and formatted output like "%s
>>>>>>>> %s" %
>>>>>>>> > (ustr, bstr). The behaviour of these is identical with CPython:
>>>>>>>> they
>>>>>>>> > raise UnicodeDecodeError because the bytes are promoted to
>>>>>>>> characters
>>>>>>>> > with a strict ascii interpretation. This happens a lot in
>>>>>>>> doctest.py and
>>>>>>>> > traceback.py, for example, where file paths and stack dumps that
>>>>>>>> include
>>>>>>>> > them, are now frequently unicode, while other inputs are byte data
>>>>>>>> > containing file paths presented in the console encoding.
>>>>>>>> >
>>>>>>>> > I can beat this into submission with enough customisation of the
>>>>>>>> stdlib
>>>>>>>> > modules, but that always makes me uncomfortable. I usually see
>>>>>>>> that as a
>>>>>>>> > hint that user code might also need to change. This may be
>>>>>>>> unfounded. I
>>>>>>>> > can probably ensure no impact to users of only ascii paths, and
>>>>>>>> the
>>>>>>>> > others seem unable to run Jython at all (in the scope of this
>>>>>>>> issue).
>>>>>>>> > However, I'm seriously wondering if I should pursue the approach
>>>>>>>> where
>>>>>>>> > file names from Java are re-encoded to bytes (maybe as utf-8
>>>>>>>> > everywhere), but that's grim.
>>>>>>>> >
>>>>>>>> > Thoughts?
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------------------------------------
>>>>>>>> ------------------
>>>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>>>> _______________________________________________
>>>>>>>> Jython-dev mailing list
>>>>>>>> Jyt...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>>>>>
>>>>>>> ------------------------------------------------------------------------------
>>>>>> Check out the vibrant tech community on one of the world's most engaging
>>>>>> tech sites, Slashdot.org! http://sdm.link/slashdot______
>>>>>> _________________________________________ Jython-dev mailing list
>>>>>> Jyt...@li... https://lists.sourceforge.net/
>>>>>> lists/listinfo/jython-dev
>>>>>>
>>>>>> ------------------------------------------------------------
>>>>>> ------------------
>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>> _______________________________________________
>>>>>> Jython-dev mailing list
>>>>>> Jyt...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>
>>>
>>>
>>> _______________________________________________
>>> Jython-dev mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>
>>>
>>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Jython-dev mailing list
>> Jyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>
>>
>