Re: [Jython-dev] Unicode user and file names (and v2.7.1)

[Javen O'N. <jav...@gm...>]

+0.5

Calling it 2.7.12 implicitly claims that it is 100% compatible with CPython
2.7.12. If you need to make a small security fix or bug fix, would it be
released as 2.7.12.1 or 2.7.13?

On May 22, 2017 10:51, "Jim Baker" <jim...@py...> wrote:

> Alan,
>
> That's a great suggestion. 2.7 was specifically chosen to show this
> correspondence. In the past, we were not so as focused on compatibility,
> but most of the changes — and corresponding delays — in what we have been
> planning to call 2.7.1 are because of the continued development on CPython
> 2.7, by backporting fixes from successive versions of CPython 3.
>
> So calling it 2.7.12 helps illustrate this. Any other thoughts on Alan's
> proposal?
>
> - Jim
>
> On Mon, May 22, 2017 at 11:25 AM, Alan Kennedy <jyt...@xh...>
> wrote:
>
>> Hi folks,
>>
>> Great to see a solid 2.7.1 jython, and work begin in earnest on jython 3.
>>
>> I have only one small suggestion to make: if jython 2.7.1 is going to be
>> one of the last 2.7 releases, maybe consider numbering it in a way that
>> indicates it is derived from the latest version of cpython 2.7.12. This
>> could indicate that it is as up-to-date as it can be, i.e. not derived from
>> cpython 2.7.1 and then abandoned.
>>
>> Perception of abandonment is often a problem for jython: I think it's
>> worth an effort to counter this mis-perception.
>>
>> Regards,
>>
>> Alan.
>>
>>
>> On Sun, May 21, 2017 at 4:35 PM, Darjus Loktevic <da...@gm...>
>> wrote:
>>
>>> Hey Guys,
>>>
>>> Regarding Jython3, looks like Isaiah has done a ton of work in 2016
>>> (CCd). Not sure how far he progressed, but indeed merging will be hard and
>>> therefore I'd say we should not diverge further while developing on both
>>> branches, but instead try to finalize 2.7 and switch to Jython3 full-time.
>>>
>>> Feel free to disagree, but here's my thinking on it:
>>>
>>>    1. Release Jython 2.7.1
>>>    2. Modernize the codebase. I think it's important for the project to
>>>    feel modern for us to attract new contributors.
>>>    1. Java8 as the minimum (may be too much for Jython2).
>>>       2. Github/core-workflow
>>>       3. (Ideally) ANTLR4 for both branches, but worst case, Jython3
>>>       only. ANTLR3 is not getting much love and ANTLR4 is quite different (does
>>>       not generate AST).
>>>       4. Gradle, directory structure.
>>>    3. Develop Jython3 primarily. Only bugfixes for 2.7 series.
>>>       1. Target 3.6 (really like the typing improvements).
>>>       2. Merge JyNI if possible.
>>>
>>> Cheers,
>>> Darjus
>>>
>>> On Sat, May 20, 2017 at 11:45 PM Jeff Allen <ja...@fa...> wrote:
>>>
>>>> Thanks all. +1 on the RC. Nearly there with my bit.
>>>>
>>>> I have fixed the  test_runpy failure James reported. It's not
>>>> Linux-specific, just I had to quieten the unlink() error to see it on
>>>> Windows. Bonus: we now pass the standard CPython test_runpy. The regrtest
>>>> has been running one last time as I typed. I've pushed to
>>>> https://bitbucket.org/tournesol/jython-utf8 just now.
>>>> I will next merge into the Jython trunk. That may not be totally smooth
>>>> because of the pervasive change. And now I think about it, it's worth a
>>>> note in NEWS. My time is a little limited today, so it could be much later
>>>> today or tomorrow evening.
>>>>
>>>> Jeff
>>>>
>>>> Jeff Allen
>>>>
>>>> On 20/05/2017 19:48, Jim Baker wrote:
>>>>
>>>> +100
>>>>
>>>> On Sat, May 20, 2017 at 12:33 PM, Darjus Loktevic <da...@gm...>
>>>> wrote:
>>>>
>>>>> Agreed regarding not blocking on 2487. That whole area needs a rewrite
>>>>> and we could potentially utilize libraries available for Java 8.
>>>>>
>>>>> Let's get Jeff's work in and do an RC?
>>>>>
>>>>> Darjus
>>>>>
>>>>> On Fri, May 19, 2017 at 7:51 PM Jim Baker <jim...@py...>
>>>>> wrote:
>>>>>
>>>>>> I don't necessarily see http://bugs.jython.org/issue2487 as a
>>>>>> blocker, but it would be nice. It just hasn't come up in real usage, unlike
>>>>>> the earlier iteration of the bug which Darjus hacked around by busy
>>>>>> waiting. I did spend some time on trying to get the publication to work
>>>>>> without racing, using the approach I detail in that bug, but no luck yet.
>>>>>> (But mostly because an utter lack of time to spend on the issue.)
>>>>>>
>>>>>> Merging in Jeff's recent work on Unicode is important and we should
>>>>>> get it in. I haven't had a chance to test myself, but given Jeff's amazing
>>>>>> attention to detail, I'm sure it's ready.
>>>>>>
>>>>>> The blocker for the RC - because we lost OSX support of setuptools
>>>>>> support of installed executables - is fixed, as I just finally confirmed:
>>>>>> http://bugs.jython.org/issue2570
>>>>>>
>>>>>>
>>>>>> On Fri, May 19, 2017 at 8:26 PM, Stefan Richthofer <
>>>>>> Ste...@gm...> wrote:
>>>>>>
>>>>>>> AFAIK every release happens by having a successful RC that is
>>>>>>> renamed to 'release' after a while. So, per definition another RC is
>>>>>>> inevitable.
>>>>>>> That said, I suppose we should get http://bugs.jython.org/issue2487
>>>>>>> fixed before we can release. I guess Jeff's work will be ready until then.
>>>>>>> At least that decision can be postponed until an RC is actually doable.
>>>>>>>
>>>>>>>
>>>>>>> *Gesendet:* Samstag, 20. Mai 2017 um 02:35 Uhr
>>>>>>> *Von:* "Darjus Loktevic" <da...@gm...>
>>>>>>> *An:* "Jeff Allen" <ja...@fa...>, "Jython Developers" <
>>>>>>> jyt...@li...>
>>>>>>> *Betreff:* Re: [Jython-dev] Unicode user and file names (and v2.7.1)
>>>>>>>
>>>>>>> Hey Jeff,
>>>>>>>
>>>>>>> Sounds good. Let's do another rc but to be honest I'm not even sure
>>>>>>> the RC matters much if there aren't people trying it except us.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>> Darjus
>>>>>>>
>>>>>>> On Fri, May 19, 2017, 1:19 AM Jeff Allen <ja...@fa...> wrote:
>>>>>>>
>>>>>>>> Hi Darjus.
>>>>>>>>
>>>>>>>> On inclusion, I'm happy to go with the community view, as always.
>>>>>>>> On one of the related tickets (http://bugs.jython.org/issue1839),
>>>>>>>> Jim said we'd get it in if timing allowed and there was some user support.
>>>>>>>>
>>>>>>>> I'm very keen to see a 2.7.1 too. The last (soft) RC was
>>>>>>>> unsuccessful, and we're still making changes, so I assume we're talking
>>>>>>>> about another RC first rather than a release?
>>>>>>>>
>>>>>>>> The UTF-8 work is nearly there, but not quite: one Linux defect to
>>>>>>>> fix, as noted on the same issue by James against the "latin-1" version.
>>>>>>>> After all the additions in the last couple of weeks (to get full BMP
>>>>>>>> support), I'm happy to find from my Linux laptop that it is still the only
>>>>>>>> thing I have to do. It looks trivial. I've been unable code at all for a
>>>>>>>> few days, so haven't looked into a solution, but now I'm back I expect to
>>>>>>>> nail it for us today or tomorrow.
>>>>>>>>
>>>>>>>> I can, of course, merge all this myself and will. I shared your
>>>>>>>> hesitancy initially, hence the fork repository, but it's turned out so well
>>>>>>>> I feel it's now low risk, as long as we still have a few days.
>>>>>>>>
>>>>>>>> I will now dive under the desk and wire up my Linux dev box.
>>>>>>>>
>>>>>>>> Jeff Allen
>>>>>>>>
>>>>>>>> On 16/05/2017 21:46, Darjus Loktevic wrote:
>>>>>>>>
>>>>>>>> Hey Jeff,
>>>>>>>>
>>>>>>>> It seems your last commit to this branch is of three days ago. Is
>>>>>>>> this ready for review? BTW, your changes look good to me.
>>>>>>>> I'm a little hesitant to merge this since we've had an RC and
>>>>>>>> REALLY have to release 2.7.1 It's miles better than 2.7.0.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Darjus
>>>>>>>>
>>>>>>>> On Mon, May 1, 2017 at 6:34 AM Jeff Allen <ja...@fa...>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I went for sys.getfilesystemencoding() == 'utf-8' and it works
>>>>>>>>> pretty
>>>>>>>>> well. Rather than just push directly I have published to here:
>>>>>>>>>
>>>>>>>>> https://bitbucket.org/tournesol/jython-utf8
>>>>>>>>>
>>>>>>>>> I write to ask for a second or third pair of eyes on it. Please
>>>>>>>>> tell me
>>>>>>>>> you can see it and whether it breaks things you care about.
>>>>>>>>>
>>>>>>>>> I touched a lot of files in the core and import system: quite a
>>>>>>>>> lot of
>>>>>>>>> tricky stuff with loaders and search paths has been adjusted. I
>>>>>>>>> think it
>>>>>>>>> a good sign that I changed hardly anything in the standard library
>>>>>>>>> we
>>>>>>>>> inherit from CPython, that we hadn't already specialised.
>>>>>>>>>
>>>>>>>>> By "works pretty well" above, I mean that the regression tests run
>>>>>>>>> cleanly for me when my user name is "Épreuve", where previously
>>>>>>>>> Jython
>>>>>>>>> died horribly. The launcher works from a Chinese user name too, as
>>>>>>>>> long
>>>>>>>>> as I localise Windows to China (CPython 2.7 feature). I can use the
>>>>>>>>> prompt and runs some tests with that setup, but I can't run the
>>>>>>>>> regression test yet, and printing a stack dump is fatal, so
>>>>>>>>> there's a
>>>>>>>>> bit more to do for Chinese.
>>>>>>>>>
>>>>>>>>> I think this means we have solid support for "latin-1" languages,
>>>>>>>>> but
>>>>>>>>> there are still places where we fatally assume bytes are Unicode
>>>>>>>>> code
>>>>>>>>> points.
>>>>>>>>>
>>>>>>>>> Jeff Allen
>>>>>>>>>
>>>>>>>>> On 05/04/2017 08:57, Jeff Allen wrote:
>>>>>>>>> > I've been working on http://bugs.jython.org/issue2356 which I'd
>>>>>>>>> like to
>>>>>>>>> > get in 2.7.1 -- it seems rather poor that Jython simply does not
>>>>>>>>> run for
>>>>>>>>> > users whose names have an un-American character ;). I know this
>>>>>>>>> issue is
>>>>>>>>> > not a blocker in most minds.
>>>>>>>>> >
>>>>>>>>> > I've made pretty good progress by allowing file names to be
>>>>>>>>> unicode
>>>>>>>>> > objects more often than they would be in CPython 2, which usually
>>>>>>>>> > returns them as bytes in some encoding that we may not know.
>>>>>>>>> I've got
>>>>>>>>> > the launcher to work properly, and straightened the logic in our
>>>>>>>>> > printing of trace-backs and exceptions from Java. Unicode file
>>>>>>>>> names
>>>>>>>>> > seems the way to go for Jython because:
>>>>>>>>> >
>>>>>>>>> >   1. Java gives us competently decoded unicode file names, from
>>>>>>>>> >      java.io.File, etc.. Re-encoding the result will be a pain
>>>>>>>>> (and
>>>>>>>>> >      overlooked).
>>>>>>>>> >   2. We appear not to have the codec we need ('mbcs'), that
>>>>>>>>> CPython
>>>>>>>>> >      reports on Windows via sys.getfilesystemencoding().
>>>>>>>>> >   3. We do this already. In 2.7.0, os.getcwd() returns unicode
>>>>>>>>> if necessary.
>>>>>>>>> >
>>>>>>>>> > Most regression tests pass. However, I'm struggling with
>>>>>>>>> test_doctest.
>>>>>>>>> > Problems arise when mixing unicode and bytes when one byte is
>>>>>>>>> 128 and
>>>>>>>>> > over. This happens in ''.join(list) and formatted output like
>>>>>>>>> "%s %s" %
>>>>>>>>> > (ustr, bstr). The behaviour of these is identical with CPython:
>>>>>>>>> they
>>>>>>>>> > raise UnicodeDecodeError because the bytes are promoted to
>>>>>>>>> characters
>>>>>>>>> > with a strict ascii interpretation. This happens a lot in
>>>>>>>>> doctest.py and
>>>>>>>>> > traceback.py, for example, where file paths and stack dumps that
>>>>>>>>> include
>>>>>>>>> > them, are now frequently unicode, while other inputs are byte
>>>>>>>>> data
>>>>>>>>> > containing file paths presented in the console encoding.
>>>>>>>>> >
>>>>>>>>> > I can beat this into submission with enough customisation of the
>>>>>>>>> stdlib
>>>>>>>>> > modules, but that always makes me uncomfortable. I usually see
>>>>>>>>> that as a
>>>>>>>>> > hint that user code might also need to change. This may be
>>>>>>>>> unfounded. I
>>>>>>>>> > can probably ensure no impact to users of only ascii paths, and
>>>>>>>>> the
>>>>>>>>> > others seem unable to run Jython at all (in the scope of this
>>>>>>>>> issue).
>>>>>>>>> > However, I'm seriously wondering if I should pursue the approach
>>>>>>>>> where
>>>>>>>>> > file names from Java are re-encoded to bytes (maybe as utf-8
>>>>>>>>> > everywhere), but that's grim.
>>>>>>>>> >
>>>>>>>>> > Thoughts?
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>>>>>>>> ------------------
>>>>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>>>>> _______________________________________________
>>>>>>>>> Jython-dev mailing list
>>>>>>>>> Jyt...@li...
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------------
>>>>>>> Check out the vibrant tech community on one of the world's most engaging
>>>>>>> tech sites, Slashdot.org! http://sdm.link/slashdot______
>>>>>>> _________________________________________ Jython-dev mailing list
>>>>>>> Jyt...@li... https://lists.sourceforge.net/
>>>>>>> lists/listinfo/jython-dev
>>>>>>>
>>>>>>> ------------------------------------------------------------
>>>>>>> ------------------
>>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>>> _______________________________________________
>>>>>>> Jython-dev mailing list
>>>>>>> Jyt...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Jython-dev mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Jython-dev mailing list
>>> Jyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>
>>>
>>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Jython-dev mailing list
> Jyt...@li...
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>
>