[Jython-dev] Summary of Jython tracker Issues

[Jython tracker <status@bu...>]

ACTIVITY SUMMARY (2017-10-06 - 2017-10-13)
Jython tracker at http://bugs.jython.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open     314 ( +1)
  closed  2335 ( +0)
  total   2649 ( +1)

Open issues with patches: 28   


Issues opened (1)
=================

#2630: jython 2.7.1 cannot import java.awt.*
http://bugs.jython.org/issue2630  opened by sergei175



Most recent 15 issues with no replies (15)
==========================================

#2618: socket.sendall no longer sends all
http://bugs.jython.org/issue2618

#2616: Incomplete / broken support for Certificate Revocation Lists
http://bugs.jython.org/issue2616

#2613: 2.7.1 MANIFEST.MF contains unfilled placeholders
http://bugs.jython.org/issue2613

#2611: mkdir() operation in /Lib/os.py has different behavior when ru
http://bugs.jython.org/issue2611

#2606: jython launch problem sun.misc.InvalidJarIndexException
http://bugs.jython.org/issue2606

#2605: Jython 2.7.0 startup performance regression
http://bugs.jython.org/issue2605

#2591: Unable to execute directory or zip file (test_cmd_line_script)
http://bugs.jython.org/issue2591

#2567: System state lost during JSR-223 initialisation
http://bugs.jython.org/issue2567

#2562: Windows: OSError: unlink(): an unknown error occurred
http://bugs.jython.org/issue2562

#2531: Support SNI for SSL/TLS server sockets
http://bugs.jython.org/issue2531

#2525: Jython incorrectly buffers file pipe output with Subprocess(co
http://bugs.jython.org/issue2525

#2520: Jython does NOT support socket.listen(0) for accepting only on
http://bugs.jython.org/issue2520

#2512: Values in built-in modules's __dict__ are “<reflected field 
http://bugs.jython.org/issue2512

#2507: Run "invokeFunction" in a thread does not inherit the "ScriptC
http://bugs.jython.org/issue2507

#2494: Support for pydoc_data
http://bugs.jython.org/issue2494



Most recent 15 issues waiting for review (15)
=============================================

#2566: inspect does not recognize code objects from bytecode files
http://bugs.jython.org/issue2566

#2545: help() does not work on Java-implemented modules
http://bugs.jython.org/issue2545

#2429: cStringIO does not work with mutable objects implementing the 
http://bugs.jython.org/issue2429

#2367: Jython ignores custom __eq__ when hashing dict subclasses
http://bugs.jython.org/issue2367

#2330: full-build fails to copy CPython License
http://bugs.jython.org/issue2330

#2230: Jython evaluation blocks under heavy load with high multi-core
http://bugs.jython.org/issue2230

#2143: site-packages support in standalone jar
http://bugs.jython.org/issue2143

#2142: Set Thread classloader when entering Jython context
http://bugs.jython.org/issue2142

#2121: Jython jar on Maven central embeds other third party libraries
http://bugs.jython.org/issue2121

#2077: marshal doesn't raise error when fed unmarshalable object
http://bugs.jython.org/issue2077

#1925: Support loading java.sql.Drivers that aren't on the boot class
http://bugs.jython.org/issue1925

#1917: No ctypes.c_char
http://bugs.jython.org/issue1917

#1866: Parser does not have mismatch token error messages caught by B
http://bugs.jython.org/issue1866

#1842: Add IBM i support to Jython
http://bugs.jython.org/issue1842

#1796: Jython doesn't support jar dir with colon's in it
http://bugs.jython.org/issue1796

Re: [Jython-dev] A new Jython DevGuide

[Jeff Allen <ja.py@fa...>]

Thanks for the thoughtful suggestion. In fact there was not so much of 
this that pulling in the text by hand was impractical. It all needs a 
human touch anyway.

J.

Jeff Allen

On 08/10/2017 16:02, Rondom@... wrote:
> On 07/10/17 09:54, Jeff Allen wrote:
>> It's not technically possible to merge the hg repo into the new git one derived from CPython's. I converted hg.python.org/jython-docs/devguide to git (using hg-fast-export), but the commits we would recognise as the same are not accepted as the same by git.
>>
>> Git merge relies on finding the common ancestor I believe. If we want to keep pulling from the CPython version, we have to derive our guide from that, visibly to git. (Not that I'd thought this through in advance.)
> Please have a look at "git replace". I have used it in the past in such situations it worked great. In my specific case I wanted to "stitch" old SVN history imported via reposurgeon to an already existing GIT-history.
>
> git-replace basically instructs git to treat one commit as equal to another one. All tools will deal with this transparently and see the new commit instead of the old commit. You might want/need to create an artificial commit to stitch the two histories together.
> Please note that the replace-refs are not pulled automatically on git fetch/clone, so it is still advisable to rewrite the history if there are not many other branches that rely on the repo.
>
>> I'll cherry-pick bits of text manually, which is more likely to be accurate than if I wrote them from scratch, and a bit quicker. Also, restoring the Mercurial instructions seems a good idea, rather than describing only a future world.
>>
>> Jeff Allen

Re: [Jython-dev] A new Jython DevGuide

[<Rondom@ro...>]

On 07/10/17 09:54, Jeff Allen wrote:
> It's not technically possible to merge the hg repo into the new git one derived from CPython's. I converted hg.python.org/jython-docs/devguide to git (using hg-fast-export), but the commits we would recognise as the same are not accepted as the same by git.
> 
> Git merge relies on finding the common ancestor I believe. If we want to keep pulling from the CPython version, we have to derive our guide from that, visibly to git. (Not that I'd thought this through in advance.)

Please have a look at "git replace". I have used it in the past in such situations it worked great. In my specific case I wanted to "stitch" old SVN history imported via reposurgeon to an already existing GIT-history.

git-replace basically instructs git to treat one commit as equal to another one. All tools will deal with this transparently and see the new commit instead of the old commit. You might want/need to create an artificial commit to stitch the two histories together.
Please note that the replace-refs are not pulled automatically on git fetch/clone, so it is still advisable to rewrite the history if there are not many other branches that rely on the repo.

> I'll cherry-pick bits of text manually, which is more likely to be accurate than if I wrote them from scratch, and a bit quicker. Also, restoring the Mercurial instructions seems a good idea, rather than describing only a future world.
> 
> Jeff Allen

Re: [Jython-dev] A new Jython DevGuide

[Jeff Allen <ja.py@fa...>]

It's not technically possible to merge the hg repo into the new git one 
derived from CPython's. I converted hg.python.org/jython-docs/devguide 
to git (using hg-fast-export), but the commits we would recognise as the 
same are not accepted as the same by git.

Git merge relies on finding the common ancestor I believe. If we want to 
keep pulling from the CPython version, we have to derive our guide from 
that, visibly to git. (Not that I'd thought this through in advance.)

I'll cherry-pick bits of text manually, which is more likely to be 
accurate than if I wrote them from scratch, and a bit quicker. Also, 
restoring the Mercurial instructions seems a good idea, rather than 
describing only a future world.

Jeff Allen

On 04/10/2017 19:27, fwierzbicki@... wrote:
> Hi Jeff,
>
> http://www.jython.org/devguide/ is produced from a fork of the CPython
> devguide - though clearly it's really out of date. The repo is here:
> https://hg.python.org/jython-docs/devguide/
>
> Thanks for looking at this!
>
> On Sat, Sep 30, 2017 at 8:59 AM, Jeff Allen <ja.py@...> wrote:
>> I created a repository in the project space on GitHub that is a clone of the
>> CPython devguide: https://github.com/jython/devguide
>>
>> The idea is to draw on the good work done recently (and still being done) by
>> CPython to define process and conventions for working on GitHub, which is
>> where, we've noted, we need to move. They've blazed a trail we'd be daft not
>> to follow wherever we can, and they have tools. In any case, that which is
>> not just process is also largely:
>>
>> Guidelines we refer to already;
>> Advice we should probably follow; and
>> Things mandatory on a PSF project.
>>
>> I'm hoping this will give us a much clearer reference for contributors, for
>> less work than maintaining those sections of the Jython wiki. If you visit
>> the CPython core developer part of the wiki it has been reduced to link into
>> the devguide. I suppose ours would go on jython.org eventually?
>>
>> I'm not sure how this will work: e.g. do we create name_jy.rst parallels;
>> can we edit mechanically; and is the Jython version on the master branch or
>> another? I've already done some deleting and re-making branches. So I'll
>> just try it and find out. At the moment I'm classifying chapters on a scale
>> of "irredeemably C-specific" through to "just as true for Jython". There are
>> not as many at the latter end as I'd hoped.
>>
>> The first outcome has been a minor contribution to the CPython version: to
>> make it build on Windows. This puts me on the contributing end of the
>> GitHub-based process as practised, which is a useful experience. I'm finding
>> review is very ... active :)
>>
>> Jeff
>>
>> --
>> Jeff Allen

[Jython-dev] Summary of Jython tracker Issues

[Jython tracker <status@bu...>]

ACTIVITY SUMMARY (2017-09-29 - 2017-10-06)
Jython tracker at http://bugs.jython.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open     313 ( +1)
  closed  2335 ( +2)
  total   2648 ( +3)

Open issues with patches: 28   


Issues opened (1)
=================

#2629: Setting javax.net.ssl.keyStore in script
http://bugs.jython.org/issue2629  opened by eknord



Most recent 15 issues with no replies (15)
==========================================

#2618: socket.sendall no longer sends all
http://bugs.jython.org/issue2618

#2616: Incomplete / broken support for Certificate Revocation Lists
http://bugs.jython.org/issue2616

#2613: 2.7.1 MANIFEST.MF contains unfilled placeholders
http://bugs.jython.org/issue2613

#2611: mkdir() operation in /Lib/os.py has different behavior when ru
http://bugs.jython.org/issue2611

#2606: jython launch problem sun.misc.InvalidJarIndexException
http://bugs.jython.org/issue2606

#2605: Jython 2.7.0 startup performance regression
http://bugs.jython.org/issue2605

#2591: Unable to execute directory or zip file (test_cmd_line_script)
http://bugs.jython.org/issue2591

#2567: System state lost during JSR-223 initialisation
http://bugs.jython.org/issue2567

#2562: Windows: OSError: unlink(): an unknown error occurred
http://bugs.jython.org/issue2562

#2531: Support SNI for SSL/TLS server sockets
http://bugs.jython.org/issue2531

#2525: Jython incorrectly buffers file pipe output with Subprocess(co
http://bugs.jython.org/issue2525

#2520: Jython does NOT support socket.listen(0) for accepting only on
http://bugs.jython.org/issue2520

#2512: Values in built-in modules's __dict__ are “<reflected field 
http://bugs.jython.org/issue2512

#2507: Run "invokeFunction" in a thread does not inherit the "ScriptC
http://bugs.jython.org/issue2507

#2494: Support for pydoc_data
http://bugs.jython.org/issue2494



Most recent 15 issues waiting for review (15)
=============================================

#2566: inspect does not recognize code objects from bytecode files
http://bugs.jython.org/issue2566

#2545: help() does not work on Java-implemented modules
http://bugs.jython.org/issue2545

#2429: cStringIO does not work with mutable objects implementing the 
http://bugs.jython.org/issue2429

#2367: Jython ignores custom __eq__ when hashing dict subclasses
http://bugs.jython.org/issue2367

#2330: full-build fails to copy CPython License
http://bugs.jython.org/issue2330

#2230: Jython evaluation blocks under heavy load with high multi-core
http://bugs.jython.org/issue2230

#2143: site-packages support in standalone jar
http://bugs.jython.org/issue2143

#2142: Set Thread classloader when entering Jython context
http://bugs.jython.org/issue2142

#2121: Jython jar on Maven central embeds other third party libraries
http://bugs.jython.org/issue2121

#2077: marshal doesn't raise error when fed unmarshalable object
http://bugs.jython.org/issue2077

#1925: Support loading java.sql.Drivers that aren't on the boot class
http://bugs.jython.org/issue1925

#1917: No ctypes.c_char
http://bugs.jython.org/issue1917

#1866: Parser does not have mismatch token error messages caught by B
http://bugs.jython.org/issue1866

#1842: Add IBM i support to Jython
http://bugs.jython.org/issue1842

#1796: Jython doesn't support jar dir with colon's in it
http://bugs.jython.org/issue1796



Issues closed (2)
=================

#2627: ⌘- AVG  technical Support phone number™ 1(800)-365+4805"
http://bugs.jython.org/issue2627  closed by stefan.richthofer

#2628: AVAST  technical Support phone number™ 1(800)-365+4805"
http://bugs.jython.org/issue2628  closed by stefan.richthofer

Re: [Jython-dev] A new Jython DevGuide

[Jeff Allen <ja.py@fa...>]

Hi Frank:

On a second look, that may have been a better starting point. You've 
already made changes in this one that I'm repeating, although I only 
recognise that because of my newly-acquired familiarity with the CPython 
text.  You've also merged from upstream (CPython) several times 
successfully whereas I've been wondering whether I can make that work at 
all.

I hadn't realised how this had moved on. I remember it being a clone of 
CPython's and I wasn't competent to help convert it in 2012. We 
should've referred each other to it more often.

The move to GitHub (as a home for it and in terms of taking CPython's 
new content) is still valid, I believe. But I'll look for a way to pull 
directly from this and try to be better at using prior art in future.

Jeff

Jeff Allen

On 04/10/2017 19:27, fwierzbicki@... wrote:
> Hi Jeff,
>
> http://www.jython.org/devguide/ is produced from a fork of the CPython
> devguide - though clearly it's really out of date. The repo is here:
> https://hg.python.org/jython-docs/devguide/
>
> Thanks for looking at this!
>
> On Sat, Sep 30, 2017 at 8:59 AM, Jeff Allen <ja.py@...> wrote:
>> I created a repository in the project space on GitHub that is a clone of the
>> CPython devguide: https://github.com/jython/devguide
>>
>> The idea is to draw on the good work done recently (and still being done) by
>> CPython to define process and conventions for working on GitHub, which is
>> where, we've noted, we need to move. They've blazed a trail we'd be daft not
>> to follow wherever we can, and they have tools. In any case, that which is
>> not just process is also largely:
>>
>> Guidelines we refer to already;
>> Advice we should probably follow; and
>> Things mandatory on a PSF project.
>>
>> I'm hoping this will give us a much clearer reference for contributors, for
>> less work than maintaining those sections of the Jython wiki. If you visit
>> the CPython core developer part of the wiki it has been reduced to link into
>> the devguide. I suppose ours would go on jython.org eventually?
>>
>> I'm not sure how this will work: e.g. do we create name_jy.rst parallels;
>> can we edit mechanically; and is the Jython version on the master branch or
>> another? I've already done some deleting and re-making branches. So I'll
>> just try it and find out. At the moment I'm classifying chapters on a scale
>> of "irredeemably C-specific" through to "just as true for Jython". There are
>> not as many at the latter end as I'd hoped.
>>
>> The first outcome has been a minor contribution to the CPython version: to
>> make it build on Windows. This puts me on the contributing end of the
>> GitHub-based process as practised, which is a useful experience. I'm finding
>> review is very ... active :)
>>
>> Jeff
>>
>> --
>> Jeff Allen

Re: [Jython-dev] A new Jython DevGuide

[<fwierzbicki@gm...>]

Hi Jeff,

http://www.jython.org/devguide/ is produced from a fork of the CPython
devguide - though clearly it's really out of date. The repo is here:
https://hg.python.org/jython-docs/devguide/

Thanks for looking at this!

On Sat, Sep 30, 2017 at 8:59 AM, Jeff Allen <ja.py@...> wrote:
> I created a repository in the project space on GitHub that is a clone of the
> CPython devguide: https://github.com/jython/devguide
>
> The idea is to draw on the good work done recently (and still being done) by
> CPython to define process and conventions for working on GitHub, which is
> where, we've noted, we need to move. They've blazed a trail we'd be daft not
> to follow wherever we can, and they have tools. In any case, that which is
> not just process is also largely:
>
> Guidelines we refer to already;
> Advice we should probably follow; and
> Things mandatory on a PSF project.
>
> I'm hoping this will give us a much clearer reference for contributors, for
> less work than maintaining those sections of the Jython wiki. If you visit
> the CPython core developer part of the wiki it has been reduced to link into
> the devguide. I suppose ours would go on jython.org eventually?
>
> I'm not sure how this will work: e.g. do we create name_jy.rst parallels;
> can we edit mechanically; and is the Jython version on the master branch or
> another? I've already done some deleting and re-making branches. So I'll
> just try it and find out. At the moment I'm classifying chapters on a scale
> of "irredeemably C-specific" through to "just as true for Jython". There are
> not as many at the latter end as I'd hoped.
>
> The first outcome has been a minor contribution to the CPython version: to
> make it build on Windows. This puts me on the contributing end of the
> GitHub-based process as practised, which is a useful experience. I'm finding
> review is very ... active :)
>
> Jeff
>
> --
> Jeff Allen

[Jython-dev] A new Jython DevGuide

[Jeff Allen <ja.py@fa...>]

I created a repository in the project space on GitHub that is a clone of 
the CPython devguide: https://github.com/jython/devguide

The idea is to draw on the good work done recently (and still being 
done) by CPython to define process and conventions for working on 
GitHub, which is where, we've noted, we need to move. They've blazed a 
trail we'd be daft not to follow wherever we can, and they have tools. 
In any case, that which is not just process is also largely:

  * Guidelines we refer to already;
  * Advice we should probably follow; and
  * Things mandatory on a PSF project.

I'm hoping this will give us a much clearer reference for contributors, 
for less work than maintaining those sections of the Jython wiki. If you 
visit the CPython core developer part of the wiki it has been reduced to 
link <https://wiki.python.org/moin/CoreDevelopment>; into the devguide. I 
suppose ours would go on jython.org eventually?

I'm not sure how this will work: e.g. do we create name_jy.rst 
parallels; can we edit mechanically; and is the Jython version on the 
master branch or another? I've already done some deleting and re-making 
branches. So I'll just try it and find out. At the moment I'm 
classifying chapters on a scale of "irredeemably C-specific" through to 
"just as true for Jython". There are not as many at the latter end as 
I'd hoped.

The first outcome has been a minor contribution to the CPython version: 
to make it build on Windows. This puts me on the contributing end of the 
GitHub-based process as practised, which is a useful experience. I'm 
finding review is very ... active :)

Jeff

-- 
Jeff Allen

[Jython-dev] Summary of Jython tracker Issues

[Jython tracker <status@bu...>]

ACTIVITY SUMMARY (2017-09-22 - 2017-09-29)
Jython tracker at http://bugs.jython.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open     312 ( -5)
  closed  2333 ( +6)
  total   2645 ( +1)

Open issues with patches: 28   


Issues opened (1)
=================

#2626: Jython 2.5 interpreter is failing due to BufferOverflow Except
http://bugs.jython.org/issue2626  opened by Megha



Most recent 15 issues with no replies (15)
==========================================

#2618: socket.sendall no longer sends all
http://bugs.jython.org/issue2618

#2616: Incomplete / broken support for Certificate Revocation Lists
http://bugs.jython.org/issue2616

#2613: 2.7.1 MANIFEST.MF contains unfilled placeholders
http://bugs.jython.org/issue2613

#2611: mkdir() operation in /Lib/os.py has different behavior when ru
http://bugs.jython.org/issue2611

#2606: jython launch problem sun.misc.InvalidJarIndexException
http://bugs.jython.org/issue2606

#2605: Jython 2.7.0 startup performance regression
http://bugs.jython.org/issue2605

#2591: Unable to execute directory or zip file (test_cmd_line_script)
http://bugs.jython.org/issue2591

#2567: System state lost during JSR-223 initialisation
http://bugs.jython.org/issue2567

#2562: Windows: OSError: unlink(): an unknown error occurred
http://bugs.jython.org/issue2562

#2531: Support SNI for SSL/TLS server sockets
http://bugs.jython.org/issue2531

#2525: Jython incorrectly buffers file pipe output with Subprocess(co
http://bugs.jython.org/issue2525

#2520: Jython does NOT support socket.listen(0) for accepting only on
http://bugs.jython.org/issue2520

#2512: Values in built-in modules's __dict__ are “<reflected field 
http://bugs.jython.org/issue2512

#2507: Run "invokeFunction" in a thread does not inherit the "ScriptC
http://bugs.jython.org/issue2507

#2494: Support for pydoc_data
http://bugs.jython.org/issue2494



Most recent 15 issues waiting for review (15)
=============================================

#2566: inspect does not recognize code objects from bytecode files
http://bugs.jython.org/issue2566

#2545: help() does not work on Java-implemented modules
http://bugs.jython.org/issue2545

#2429: cStringIO does not work with mutable objects implementing the 
http://bugs.jython.org/issue2429

#2367: Jython ignores custom __eq__ when hashing dict subclasses
http://bugs.jython.org/issue2367

#2330: full-build fails to copy CPython License
http://bugs.jython.org/issue2330

#2230: Jython evaluation blocks under heavy load with high multi-core
http://bugs.jython.org/issue2230

#2143: site-packages support in standalone jar
http://bugs.jython.org/issue2143

#2142: Set Thread classloader when entering Jython context
http://bugs.jython.org/issue2142

#2121: Jython jar on Maven central embeds other third party libraries
http://bugs.jython.org/issue2121

#2077: marshal doesn't raise error when fed unmarshalable object
http://bugs.jython.org/issue2077

#1925: Support loading java.sql.Drivers that aren't on the boot class
http://bugs.jython.org/issue1925

#1917: No ctypes.c_char
http://bugs.jython.org/issue1917

#1866: Parser does not have mismatch token error messages caught by B
http://bugs.jython.org/issue1866

#1842: Add IBM i support to Jython
http://bugs.jython.org/issue1842

#1796: Jython doesn't support jar dir with colon's in it
http://bugs.jython.org/issue1796



Top 10 most discussed issues (1)
================================

#2626: Jython 2.5 interpreter is failing due to BufferOverflow Except
http://bugs.jython.org/issue2626   5 msgs



Issues closed (1)
=================

#2476: Functions reflected from Java are missing attributes (used to 
http://bugs.jython.org/issue2476  closed by zyasoft

Re: [Jython-dev] A word about Mercurial commit messages

[Jeff Allen <ja.py@fa...>]

I notice the CPython devguide -- therefore the future Jython devguide(!) 
-- has similar advice 
(https://devguide.python.org/committing#commit-messages) to Frank's 
recommendation below: pithy one-liner, blank line, complete brief 
information.

It refrains from specifying actual line lengths, but 72 has been working 
for me. (Never sure whether to line-break paragraph #2+ manually.)

And GitHub is unforgiving after 72: 
https://github.com/jython/jython3/commit/5b4dc2d54d01a6fda8c55d07b2608167e7a40769, 
so I guess we'll have to fit in.

Jeff


On 12/09/2014 07:21, Jeff Allen wrote:
> I'm happy to follow Frank. Scrolling back down Hg Workbench I see that 
> (lately) he, Indra and Santoso to varying degrees follow the 
> recommended convention. (I'm not sure I can stick to 50 for the 
> summary!) Hg Workbench gives me a settable red line, by default at 
> column 80 (but 72 if we follow the blog). I seem to have wrap and EOL 
> options, but they don't stick :(.
>
> Jeff
> Jeff Allen
> On 11/09/2014 23:47, Jim Baker wrote:
>>
>> +1, as a frequent violator myself. I plan to reform my ways ;), since 
>> this makes a lot of sense for seeing the summaries in the abstract, 
>> vs say just reviewing hg log.
>>
>>
>> -- 
>> - Jim
>> Jim Baker
>>
>>
>> On September 11, 2014 at 3:44:06 PM, fwierzbicki@... 
>> (fwierzbicki@... <mailto:fwierzbicki@...>) wrote:
>>
>>> On Thu, Sep 11, 2014 at 1:23 PM, Jeff Allen <ja.py@...> wrote:
>>> > Mercurial will use your commit message up to the first line break 
>>> as the
>>> > abstract it displays, for example here: http://hg.python.org/jython/
>>> > Worth thinking about how that will read. I don't think we have an 
>>> actual
>>> > rule about length.
>>> Although I don't always practice it, I like the guideline that I've
>>> seen in git circles detailed here:
>>> http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
>>>
>>> The important part being a 50 char or less first line separated by a
>>> blank line, followed by the detailed message if necessary.
>>>
>>> -Frank
>>>
>

Re: [Jython-dev] [Infrastructure-staff] Errors when pushing to hg.python.org/jython

[Jeff Allen <ja.py@fa...>]

Thanks Ernest!

First observed for me last night: Mon, 25 Sep 2017 about 21:00 +0100, 
but I haven't pushed for about 4 weeks (been working in a forked repo). 
I started this thread almost immediately after the merge and push 
(https://hg.python.org/jython/rev/50393fac6f20).

My push 4 weeks ago (/Sun Aug 27 07:45:40 EDT 2017/) is the last thing 
on the archive 
https://mail.python.org/pipermail/jython-checkins/2017-August/date.html 
and I remember no such an error.

Since Jim says  "I have also seen this error.  ..." I assume that would 
have been when pushing soon after Sat, 09 Sep 2017 14:02:01 -0600 
(https://hg.python.org/jython/rev/8d52ad2559f5). However, he could be 
more precise.

So on or after 27 Aug and (maybe) before 10 Sep?

Jeff


On 26/09/2017 00:49, Ernest W. Durbin III wrote:
> Hey!
>
> Ernest here! I’m one of the volunteers on the PSF infra staff.
>
> Yeah it is fair to say that the mercurial installation has not gotten 
> much love since the core python workflows moved to GitHub.
> But we also haven’t heard any complaints until now.
>
> I’m AFK right now but will be able to spend some time trying to help 
> in the morning.
>
> Was this a recent failure for y’all or has it been ongoing? Knowing 
> when it went goofy will be a big help!
>
> -Ernest
>
> On September 25, 2017 at 6:13:46 PM, Jeff Allen (ja.py@... 
> <mailto:ja.py@...>) wrote:
>
>> Or administered on the assumption everyone has moved Git. 
>> https://hg.python.org/?sort=lastchange
>>
>> I think this (fairly obviously) is the hook that sends e-mail, but 
>> why the call has changed I can't guess.
>>
>> I'm happy to move to GitHub. Start with the CPython dev guide as it 
>> contains a well thought-out way of working? (I keep meaning to.)
>>
>> Jeff
>>
>>
>> On 25/09/2017 21:40, Jim Baker wrote:
>>> I have also seen this error. I assume that a post-commit hook is 
>>> being run on the hg server, and it is failing. Not good.
>>>
>>> This message may be implicitly telling us we need to get off 
>>> hg.python.org <http://hg.python.org>; sooner than later, since it's 
>>> no longer being as actively administered. (Perhaps not at all in fact.)
>>>
>>> On Mon, Sep 25, 2017 at 2:35 PM, Stefan Richthofer 
>>> <Stefan.Richthofer@... <mailto:Stefan.Richthofer@...>> wrote:
>>>
>>>     Also observed these errors and additionally I did not receive an
>>>     email notification from jython-checkins.
>>>     Not that I cared too much for the email notification (still nice
>>>     to have), but I suspect the issues could be
>>>     related. Or did you receive email notification for recent pushes?
>>>     *Gesendet:* Montag, 25. September 2017 um 22:20 Uhr
>>>     *Von:* "Jeff Allen" <ja.py@... <mailto:ja.py@...>>
>>>     *An:* "Jython Developers" <jython-dev@...
>>>     <mailto:jython-dev@...>>,
>>>     infrastructure-staff@...
>>>     <mailto:infrastructure-staff@...>
>>>     *Betreff:* [Jython-dev] Errors when pushing to
>>>     hg.python.org/jython <http://hg.python.org/jython>;
>>>
>>>     Just now got these error messages, actually 4 repeats, one for
>>>     each change set:
>>>
>>>     PS jython-int> hg push
>>>     pushing to ssh://hg@.../jython
>>>     <mailto:ssh://hg@.../jython>;
>>>     searching for changes
>>>     remote: adding changesets
>>>     remote: adding manifests
>>>     remote: adding file changes
>>>     remote: added 4 changesets with 28 changes to 25 files
>>>     remote: Traceback (most recent call last):
>>>     remote:   File "/srv/hg/repos/hooks/mail.py", line 166, in incoming
>>>     remote:     return _incoming(ui, repo, **kwargs)
>>>     remote:   File "/srv/hg/repos/hooks/mail.py", line 102, in _incoming
>>>     remote:     diffstat = patch.diffstat(iterlines(diffchunks),
>>>     width=60, git=True)
>>>     remote: TypeError: diffstat() got an unexpected keyword argument
>>>     'git'
>>>     remote: error: incoming.notify hook raised an exception:
>>>     diffstat() got an unexpected keyword argument 'git'
>>>     remote: (run with --traceback for stack trace)
>>>
>>>     ... and 3 more the same (from "remote: Traceback ...").
>>>
>>>     It seems to have worked (my change sets landed), but something's
>>>     not right at the PSF end. Any ideas?
>>>
>>>     Jeff
>>>
>>>     ------------------------------------------------------------------------------
>>>     Check out the vibrant tech community on one of the world's most
>>>     engaging tech sites, Slashdot.org!
>>>     http://sdm.link/slashdot_______________________________________________
>>>     <http://sdm.link/slashdot_______________________________________________>;
>>>     Jython-dev mailing list Jython-dev@...
>>>     <mailto:Jython-dev@...>
>>>     https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>     <https://lists.sourceforge.net/lists/listinfo/jython-dev>;
>>>
>>>     ------------------------------------------------------------------------------
>>>     Check out the vibrant tech community on one of the world's most
>>>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>     _______________________________________________
>>>     Jython-dev mailing list
>>>     Jython-dev@...
>>>     <mailto:Jython-dev@...>
>>>     https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>     <https://lists.sourceforge.net/lists/listinfo/jython-dev>;
>>>
>>>
>>
>> _______________________________________________
>> Infrastructure-staff mailing list
>> Infrastructure-staff@... <mailto:Infrastructure-staff@...>
>> https://mail.python.org/mailman/listinfo/infrastructure-staff

[Jython-dev] Java9 dynalink

[Darjus Loktevic <darjus@gm...>]

http://docs.oracle.com/javase/9/docs/api/jdk/dynalink/DynamicLinker.html
Looks useful :)

Darjus

Re: [Jython-dev] PyType publication with better concurrency

[Darjus Loktevic <darjus@gm...>]

Nice!

On Mon, Sep 25, 2017 at 4:13 AM Jeff Allen <ja.py@...> wrote:

> Thanks Jim. I'm working on presenting this as a reduced number of commits
> to make the change easier to follow.
>
> I'll strip out the debug too. After so many weeks, I shall miss it, but
> it's too in-your-face. If we need it again, it will be on Bitbucket for a
> while.
> +1 for an early 2.7.2: we should make a short list of blockers on its own
> thread.
>
>
> Jeff
>
>
> On 24/09/2017 17:18, Jim Baker wrote:
>
> Likewise, glad to see it was just a problem in the test setup. Everything
> looks good now: the test demonstrates that the Python type mapping does not
> prevent class GC, assuming nothing else refers to the Java class. Indeed
> the earlier faulty version of the test further demonstrates that point,
> given the parent classloader referring to the class, and preventing GC.
>
> I have also had a chance to review the rest of the change set, and
> everything else looks great. In particular, I like how you have removed the
> need to support the hack for hardRef, along with the management of deferred
> initialization of types.
>
> Let's merge this code in, given that it's a substantial improvement over
> the old implementation, with all tests passing. We should consider
> releasing a 2.7.2 soon, since this and related changes has fixed serious
> problems in re and other mixed usage of Java and Python via callbacks (
> http://bugs.jython.org/issue2609). Getting this out would then allow us
> to work on Java 9 support in 2.7.3, as well as fixing up sys slowness and
> related jsr223 (hopefully!).
>
> - Jim
>
>
>
> On Sun, Sep 24, 2017 at 3:19 AM, Jeff Allen <ja.py@...> wrote:
>
>> As I suspected, the critical distinction is between the *initiating*
>> class loader and the *defining* class loader.
>>
>> The defining class loader here turns out to be an instance of
>> sun.misc.Launcher$AppClassLoader, the grandparent of our URLClassLoader
>> that gets consulted before it tries the JAR URL we gave.
>> org.python.tests.Callbacker appears in jython-dev.jar, so it is found there
>> first. This is the same loader that holds PyObject, etc., so essentially
>> permanent and keeps our test class alive.
>>
>> When this is all fixed by making the parent explicitly None in
>> make_jar_classloader, everything is GC'd as we'd hope. (Phew!)
>>
>>
>> https://bitbucket.org/tournesol/jython-fgtype/commits/4fd2baafb7499929f876ac20a607511668b3e222
>> Jeff
>>
>>
>>
>> On 18/09/2017 00:45, Jim Baker wrote:
>>
>> Jeff,
>>
>> I looked at the new test, and it's interesting to see the ClassLoader
>> gets collected, but not the corresponding PyJavaType. To me this implies
>> the Class was necessarily collected; but it's not observable directly in
>> Python code. While you are busy on other things, I will try playing with
>> this changeset some more to see what is going on here. Maybe it ends just
>> being a simple question of setting up a ref queue on Class GC and calling
>> ClassValue#remove.
>>
>> It's rather awesome about the simplification of fromClass, and removing
>> the separate implementation of fromClassSkippingInners.
>>
>> - Jim
>>
>> On Sun, Sep 17, 2017 at 2:41 AM, Jeff Allen <ja.py@...> wrote:
>>
>>> Jim:
>>>
>>> I figured out needsInners (I think!) and how I could make fromClass and
>>> fromClassSkippingInners into one method using the state of the registry to
>>> steer the action to be almost the same -- and reach the same conclusion --
>>> with a net reduction in code.
>>>
>>>
>>> https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e
>>>
>>> I have added a test to prove the PyType (actually PyJavaType) is GC'd
>>> when we cease to reference it, using make_jar_classloader, but it is
>>> presently failing. Applied retrospectively to successive change sets it
>>> begins to fail exactly where I introduce ClassValue. :(
>>> Maybe ClassValue doesn't quite work as I thought. Or I'm still not using
>>> it as John Rose intended. It seems wrong we should have to layer weak
>>> references on top or have some kind of removal queue.
>>>
>>> I tried visualvm to see where it might be referenced, but I'm not sure
>>> how to interpret what it's telling me. However, this week, and maybe
>>> longer, other things have priority. Ideas welcome for when I'm back to this.
>>> For exposed PyObjects, I'm pretty sure classToBuilder always was keeping
>>> a strong reference that would keep them alive now. (But not for the type
>>> under test.) I've removed this effect. But maybe we don't mind that?
>>>
>>> https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e#Lsrc/org/python/core/PyType.javaT1208
>>>
>>> Jeff
>>>
>>>
>>> On 12/09/2017 01:37, Jim Baker wrote:
>>>
>>> [I'm been having some problem posting to the jython-dev mailing list,
>>> let's see if this response gets through!]
>>>
>>> On Mon, Sep 11, 2017 at 1:47 PM, Jeff Allen <ja.py@...> wrote:
>>>
>>>> Jim:
>>>>
>>>> Thanks for the pointers. I agree with you on #2 that what I've got
>>>> *should* behave correctly with class loaders, but is there a test you can
>>>> suggest (or one I've missed) that conclusively demonstrates it?
>>>>
>>>
>>> So I would start with test_support.make_jar_classloader, just because
>>> it's easier to test this behavior in Python. test_classpathimporter seems
>>> to use this support functionality nicely.
>>>
>>> If we want to add testing of our expectations around ClassValue's
>>> lifetime, we can also add some sort of weakref scheme, in conjunction with
>>> a multi-round GC, to verify that the PyType gets collected as expected. Of
>>> course we should expect ClassValue to behave as expected, but this would
>>> ensure we are not holding anything, now or in the future implementation, in
>>> the Jython runtime that would violate this contract.
>>>
>>>
>>>> I'll press in on the question of inners. I tried just calling
>>>> fromClass, but it fails when the inner extends the outer (test_import_jy::
>>>> test_selfreferential_classes fails). You have to initialise the outer, then
>>>> visit the inners. In a way, it's the PyJavaType counterpart of the deferred
>>>> initialisation of exposed types that we control by identifying bootstrap
>>>> types. But here it has to be self-guiding.
>>>>
>>>> I have an idea that if I can make fromClass and fromClassSkippingInners
>>>> the same thing, then that can become the implementation
>>>> classToValue.computeValue, and I won't need the NotReady exception or
>>>> findExistingType.
>>>>
>>>
>>> So Java classes can form a general type graph (directed, with cycles).
>>> It seems to me that the NotReady acts as some sort of coloring in the graph
>>> exploration that fromClass is doing. But as I said, I find this code for
>>> inner classes rather subtle! Good luck in your coding here, it would be
>>> really nice to get some further simplification.
>>>
>>>
>>>>
>>>> Jeff
>>>>
>>>> On 10/09/2017 01:12, Jim Baker wrote:
>>>>
>>>>> Thanks for working on this change set, it's really good stuff. In
>>>>> particular, that's a very nice handoff you do in the PyType.Registry class,
>>>>> specifically moving computed types from classToNewType to classToType; and
>>>>> managing the NotReady state. Perhaps it needs some sort of state diagram to
>>>>> illustrate, but as someone who has been in that code, it makes perfect
>>>>> sense.
>>>>>
>>>>> With respect to GC, the following should be the case:
>>>>>
>>>>>   * As long as the ClassLoader for the mapped Java class instance C is
>>>>>     around, the corresponding PyType instance T will live. This
>>>>>     follows from the definition of ClassValue as you point out in the
>>>>>     code comments. (Good!)
>>>>>   * As long as Python code holds a hard ref to T, the corresponding C
>>>>>     will live. (Good!) (So we would see this in from `some_java_pkg
>>>>>     import baz`; and it follows from the underlying_class field
>>>>>     holding a hard ref from T to C.)
>>>>>   * As long as Python code holds a hard ref to an instance of C, the
>>>>>     corresponding T will also live. (Good!)
>>>>>
>>>>> Otherwise, this mapping will allow for GC of T, and subsequently of C,
>>>>> per the definition of ClassValue. I was also looking at Groovy's usage of a
>>>>> similar mapping, and corresponding bug,
>>>>> https://bugs.openjdk.java.net/browse/JDK-8136353, and it looks like
>>>>> the problem reported there doesn't hold. This is because there are no
>>>>> strong refs outside of PyType itself.
>>>>>
>>>>> So everything looks good for getting this change in.
>>>>>
>>>>> On Sat, Sep 9, 2017 at 2:12 PM, Jeff Allen <ja.py@...
>>>>> <mailto:ja.py@...>> wrote:
>>>>>
>>>>>     Jim, all:
>>>>>
>>>>>     By a series of small steps I've come quite a long way in this
>>>>>     aspect of PyType. It might be worth a look.
>>>>>
>>>>> https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd
>>>>>     <
>>>>> https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd
>>>>> >
>>>>>
>>>>>     In this last commit I've entirely dispensed with any long-term
>>>>>     reference to the created PyType in the type registry: only the
>>>>>     ClassValue holds one. The explicit management of reachability goes
>>>>>     away and we have just the behaviour given us by ClassValue. The
>>>>>     only down-side I see is that a PyType created for a Java class you
>>>>>     handle briefly from Python will not be collectable until the class
>>>>> is.
>>>>>
>>>>>     The old test for weakness, which I've been running in parallel
>>>>>     with the new, requires PyType.getClassToType(), and can no longer
>>>>>     be supported. (You can't enumerate the ClassValue.) I believe the
>>>>>     replacement tests the same thing -- they used to pass and fail
>>>>>     together.
>>>>>
>>>>>
>>>>> Right, it makes sense to remove this test. This is because we don't
>>>>> want to support the necessary mapping, even though it could be readily
>>>>> supported with an appropriate concurrent weak mapping - the only semantics
>>>>> it would be supporting is for this very test.
>>>>>
>>>>>
>>>>>     I've moved some actions around, and removed some of the checks. We
>>>>>     seemingly expected the desired PyType to be created at any moment
>>>>>     by a side effect. Reason and regression testing tell me that
>>>>>     doesn't happen in the new arrangement. I think there is slightly
>>>>>     less code to execute, but masses more comment and debug -- it's my
>>>>>     way of figuring it out. It's all very educational.
>>>>>
>>>>>     I still don't understand:
>>>>>
>>>>>     1. Whether we find the downside (PyType not GC'd until the class
>>>>> is)
>>>>>        important.
>>>>>
>>>>>
>>>>> So we do know that Jython will continue to work if we remove a mapping
>>>>> entry, since it happens all the time currently in some code like json, re,
>>>>> and socket, where classes are not directly imported :) We could always do
>>>>> some sort of LRU and then call ClassValue#remove, but I don't think it
>>>>> makes sense. There are many other other optimizations in the runtime first.
>>>>>
>>>>>     2. Whether I have broken something that the regression tests don't
>>>>>        cover (like use with isolating class loaders).
>>>>>
>>>>>
>>>>> These are different Java classes if in different class loaders. For
>>>>> Python usage of these Java classes, it's all the same, thanks to duck
>>>>> typing, but we need to maintain a different mapping entry. Again #1
>>>>> demonstrates we have been handling this correctly in the current codebase.
>>>>>
>>>>>     3. What needsInners is really up to. I see *what* it does, but not
>>>>>        *why* well enough to consider alternatives. It would simplify
>>>>>        computeValue if there were no fromClassSkippingInners: could
>>>>>        needsInners be a registry global? And why do we skip processing
>>>>>        them, when we do?
>>>>>
>>>>>
>>>>> That codepath is exceedingly subtle, at least to me. But it would be
>>>>> an interesting experiment to see if it could be made global, vs this
>>>>> potentially expanding context that is passed through.
>>>>>
>>>>> - Jim
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Jython-dev mailing list
> Jython-dev@...
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>

Re: [Jython-dev] Errors when pushing to hg.python.org/jython

[Jeff Allen <ja.py@fa...>]

Or administered on the assumption everyone has moved Git. 
https://hg.python.org/?sort=lastchange

I think this (fairly obviously) is the hook that sends e-mail, but why 
the call has changed I can't guess.

I'm happy to move to GitHub. Start with the CPython dev guide as it 
contains a well thought-out way of working? (I keep meaning to.)

Jeff


On 25/09/2017 21:40, Jim Baker wrote:
> I have also seen this error. I assume that a post-commit hook is being 
> run on the hg server, and it is failing. Not good.
>
> This message may be implicitly telling us we need to get off 
> hg.python.org <http://hg.python.org>; sooner than later, since it's no 
> longer being as actively administered. (Perhaps not at all in fact.)
>
> On Mon, Sep 25, 2017 at 2:35 PM, Stefan Richthofer 
> <Stefan.Richthofer@... <mailto:Stefan.Richthofer@...>> wrote:
>
>     Also observed these errors and additionally I did not receive an
>     email notification from jython-checkins.
>     Not that I cared too much for the email notification (still nice
>     to have), but I suspect the issues could be
>     related. Or did you receive email notification for recent pushes?
>     *Gesendet:* Montag, 25. September 2017 um 22:20 Uhr
>     *Von:* "Jeff Allen" <ja.py@... <mailto:ja.py@...>>
>     *An:* "Jython Developers" <jython-dev@...
>     <mailto:jython-dev@...>>,
>     infrastructure-staff@...
>     <mailto:infrastructure-staff@...>
>     *Betreff:* [Jython-dev] Errors when pushing to
>     hg.python.org/jython <http://hg.python.org/jython>;
>
>     Just now got these error messages, actually 4 repeats, one for
>     each change set:
>
>     PS jython-int> hg push
>     pushing to ssh://hg@.../jython
>     <mailto:ssh://hg@.../jython>;
>     searching for changes
>     remote: adding changesets
>     remote: adding manifests
>     remote: adding file changes
>     remote: added 4 changesets with 28 changes to 25 files
>     remote: Traceback (most recent call last):
>     remote:   File "/srv/hg/repos/hooks/mail.py", line 166, in incoming
>     remote:     return _incoming(ui, repo, **kwargs)
>     remote:   File "/srv/hg/repos/hooks/mail.py", line 102, in _incoming
>     remote:     diffstat = patch.diffstat(iterlines(diffchunks),
>     width=60, git=True)
>     remote: TypeError: diffstat() got an unexpected keyword argument 'git'
>     remote: error: incoming.notify hook raised an exception:
>     diffstat() got an unexpected keyword argument 'git'
>     remote: (run with --traceback for stack trace)
>
>     ... and 3 more the same (from "remote: Traceback ...").
>
>     It seems to have worked (my change sets landed), but something's
>     not right at the PSF end. Any ideas?
>
>     Jeff
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org!
>     http://sdm.link/slashdot_______________________________________________
>     <http://sdm.link/slashdot_______________________________________________>;
>     Jython-dev mailing list Jython-dev@...
>     <mailto:Jython-dev@...>
>     https://lists.sourceforge.net/lists/listinfo/jython-dev
>     <https://lists.sourceforge.net/lists/listinfo/jython-dev>;
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>     _______________________________________________
>     Jython-dev mailing list
>     Jython-dev@...
>     <mailto:Jython-dev@...>
>     https://lists.sourceforge.net/lists/listinfo/jython-dev
>     <https://lists.sourceforge.net/lists/listinfo/jython-dev>;
>
>

[Jython-dev] Refined type hierarchy of dictionary iterators

[Stefan Richthofer <Stefan.Richthofer@gm...>]

Hi all,

I refined dicitonary iterator types a bit:
https://github.com/jythontools/jython/commit/5b1ed9b0e42dcaa21549ea4a10ee9136c1fb426e

Goal is to allow an easier support for iterator types in JyNI
(cf. https://github.com/Stewori/JyNI/issues/13).

This is fairly internal stuff and does not affect public API, so it shouldn't break
anything that is "properly" using Jython API. However, we all know that internal API
sometimes is not as internal as one might expect in Java. So, if somebody sees an
issue with this, speak up now ;)

Main design goal of this change:

- Let iterators for values, keys and items be of distinct classes
  (because CPython has it this way and then it's easier to match in JyNI)
- Let the same kind of such iterators from PyDictionary and PyStringMap
  have a common superclass (defined in AbstractDict)

Best,

-Stefan

Re: [Jython-dev] Errors when pushing to hg.python.org/jython

[Jim Baker <jim.baker@py...>]

I have also seen this error. I assume that a post-commit hook is being run
on the hg server, and it is failing. Not good.

This message may be implicitly telling us we need to get off hg.python.org
sooner than later, since it's no longer being as actively administered.
(Perhaps not at all in fact.)

On Mon, Sep 25, 2017 at 2:35 PM, Stefan Richthofer <Stefan.Richthofer@...
> wrote:

> Also observed these errors and additionally I did not receive an email
> notification from jython-checkins.
> Not that I cared too much for the email notification (still nice to have),
> but I suspect the issues could be
> related. Or did you receive email notification for recent pushes?
>
>
> *Gesendet:* Montag, 25. September 2017 um 22:20 Uhr
> *Von:* "Jeff Allen" <ja.py@...>
> *An:* "Jython Developers" <jython-dev@...>,
> infrastructure-staff@...
> *Betreff:* [Jython-dev] Errors when pushing to hg.python.org/jython
>
> Just now got these error messages, actually 4 repeats, one for each change
> set:
>
> PS jython-int> hg push
> pushing to ssh://hg@.../jython
> searching for changes
> remote: adding changesets
> remote: adding manifests
> remote: adding file changes
> remote: added 4 changesets with 28 changes to 25 files
> remote: Traceback (most recent call last):
> remote:   File "/srv/hg/repos/hooks/mail.py", line 166, in incoming
> remote:     return _incoming(ui, repo, **kwargs)
> remote:   File "/srv/hg/repos/hooks/mail.py", line 102, in _incoming
> remote:     diffstat = patch.diffstat(iterlines(diffchunks), width=60,
> git=True)
> remote: TypeError: diffstat() got an unexpected keyword argument 'git'
> remote: error: incoming.notify hook raised an exception: diffstat() got an
> unexpected keyword argument 'git'
> remote: (run with --traceback for stack trace)
>
> ... and 3 more the same (from "remote: Traceback ...").
>
> It seems to have worked (my change sets landed), but something's not right
> at the PSF end. Any ideas?
>
> Jeff
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most engaging
> tech sites, Slashdot.org! http://sdm.link/slashdot______
> _________________________________________ Jython-dev mailing list
> Jython-dev@... https://lists.sourceforge.net/
> lists/listinfo/jython-dev
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Jython-dev mailing list
> Jython-dev@...
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>
>

Re: [Jython-dev] Errors when pushing to hg.python.org/jython

[Stefan Richthofer <Stefan.Richthofer@gm...>]

<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>
<div>Also observed these errors and additionally I did not receive an email notification from jython-checkins.</div>

<div>Not that I cared too much for the email notification (still nice to have), but I suspect the issues could be</div>

<div>related. Or did you receive email notification for recent pushes?</div>

<div>&nbsp;</div>

<div>&nbsp;
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"><b>Gesendet:</b>&nbsp;Montag, 25. September 2017 um 22:20 Uhr<br/>
<b>Von:</b>&nbsp;&quot;Jeff Allen&quot; &lt;ja.py@...&gt;<br/>
<b>An:</b>&nbsp;&quot;Jython Developers&quot; &lt;jython-dev@...&gt;, infrastructure-staff@...<br/>
<b>Betreff:</b>&nbsp;[Jython-dev] Errors when pushing to hg.python.org/jython</div>

<div name="quoted-content">
<div style="background-color: rgb(255,255,255);">
<p>Just now got these error messages, actually 4 repeats, one for each change set:</p>

<p>PS jython-int&gt; hg push<br/>
pushing to <a class="moz-txt-link-abbreviated" href="mailto:ssh://hg@.../jython"; onclick="parent.window.location.href=&#39;ssh:&#92;/&#92;/hg@...&#92;/jython&#39;; return false;" target="_blank">ssh://hg@.../jython</a><br/>;
searching for changes<br/>
remote: adding changesets<br/>
remote: adding manifests<br/>
remote: adding file changes<br/>
remote: added 4 changesets with 28 changes to 25 files<br/>
remote: Traceback (most recent call last):<br/>
remote:&nbsp;&nbsp; File &quot;/srv/hg/repos/hooks/mail.py&quot;, line 166, in incoming<br/>
remote:&nbsp;&nbsp;&nbsp;&nbsp; return _incoming(ui, repo, **kwargs)<br/>
remote:&nbsp;&nbsp; File &quot;/srv/hg/repos/hooks/mail.py&quot;, line 102, in _incoming<br/>
remote:&nbsp;&nbsp;&nbsp;&nbsp; diffstat = patch.diffstat(iterlines(diffchunks), width=60, git=True)<br/>
remote: TypeError: diffstat() got an unexpected keyword argument &#39;git&#39;<br/>
remote: error: incoming.notify hook raised an exception: diffstat() got an unexpected keyword argument &#39;git&#39;<br/>
remote: (run with --traceback for stack trace)</p>

<p>... and 3 more the same (from &quot;remote: Traceback ...&quot;).</p>

<p>It seems to have worked (my change sets landed), but something&#39;s not right at the PSF end. Any ideas?</p>

<p>Jeff</p>
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world&#39;s most engaging tech sites, Slashdot.org! <a href="http://sdm.link/slashdot_______________________________________________"; target="_blank">http://sdm.link/slashdot_______________________________________________</a>; Jython-dev mailing list Jython-dev@... <a href="https://lists.sourceforge.net/lists/listinfo/jython-dev"; target="_blank">https://lists.sourceforge.net/lists/listinfo/jython-dev</a></div>;
</div>
</div>
</div>
</div></div></body></html>

[Jython-dev] Errors when pushing to hg.python.org/jython

[Jeff Allen <ja.py@fa...>]

Just now got these error messages, actually 4 repeats, one for each 
change set:

PS jython-int> hg push
pushing to ssh://hg@.../jython
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 4 changesets with 28 changes to 25 files
remote: Traceback (most recent call last):
remote:   File "/srv/hg/repos/hooks/mail.py", line 166, in incoming
remote:     return _incoming(ui, repo, **kwargs)
remote:   File "/srv/hg/repos/hooks/mail.py", line 102, in _incoming
remote:     diffstat = patch.diffstat(iterlines(diffchunks), width=60, 
git=True)
remote: TypeError: diffstat() got an unexpected keyword argument 'git'
remote: error: incoming.notify hook raised an exception: diffstat() got 
an unexpected keyword argument 'git'
remote: (run with --traceback for stack trace)

... and 3 more the same (from "remote: Traceback ...").

It seems to have worked (my change sets landed), but something's not 
right at the PSF end. Any ideas?

Jeff

Re: [Jython-dev] JyNI 2.7-alpha.5 released: JyNI also runs on Windows now

[Jeff Allen <ja.py@fa...>]

Stefan:

Sincere felicitations on your progress this GSoC. I think we all know 
people who would love to use C extensions with Jython and we can't 
rewrite them all in Java.

Jeff


On 23/09/2017 18:03, Stefan Richthofer wrote:
> With JyNI alpha 5, I proudly announce the first version of JyNI that
> also runs on Windows.
>
> JyNI is a layer that allows Jython to load native CPython extensions.
> Most notably ctypes and NumPy are already workable (core functionality).
>
>
> Take a look at
>
> jyni.org
>
> or go directly to the release:
>
> https://github.com/Stewori/JyNI/releases/tag/v2.7-alpha.5
>
>
> Binaries are provided for Linux, OS X and Windows.
> It should in principle also work on other POSIX platforms.
>
>
>
> NumPy support
> -------------
>
> >From JyNI alpha 4 onwards, some NumPy support is available.
> For JyNI alpha 5 featuring Windows support, we explicitly asserted that
> NumPy works on Windows equally well as on Linux and OS X,
> i.e. up to known limitations.
>
> For details about NumPy support see
>
> https://github.com/Stewori/JyNI/releases/tag/v2.7-alpha.4
> https://github.com/Stewori/JyNI/issues/2
>
>
>
> Google Summer of Code
> ---------------------
>
> JyNI alpha 5 was mainly developed within a Google Summer of Code project:
>
> https://summerofcode.withgoogle.com/projects/#4931062149939200
> http://gsoc2017-jyni.blogspot.de for details.
>
>
>
> What comes next?
> ----------------
>
> The Python C API is not yet fully supported. A major goal for the next
> release is to add support for the buffer protocol, which is currently
> blocking support for SciPy and various other extensions.
>
>
>
> Enjoy!
>
>
> -Stefan
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Jython-dev mailing list
> Jython-dev@...
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>

Re: [Jython-dev] PyType publication with better concurrency

[Jeff Allen <ja.py@fa...>]

Thanks Jim. I'm working on presenting this as a reduced number of 
commits to make the change easier to follow.

I'll strip out the debug too. After so many weeks, I shall miss it, but 
it's too in-your-face. If we need it again, it will be on Bitbucket for 
a while.

+1 for an early 2.7.2: we should make a short list of blockers on its 
own thread.

Jeff

On 24/09/2017 17:18, Jim Baker wrote:
> Likewise, glad to see it was just a problem in the test setup. 
> Everything looks good now: the test demonstrates that the Python type 
> mapping does not prevent class GC, assuming nothing else refers to the 
> Java class. Indeed the earlier faulty version of the test further 
> demonstrates that point, given the parent classloader referring to the 
> class, and preventing GC.
>
> I have also had a chance to review the rest of the change set, and 
> everything else looks great. In particular, I like how you have 
> removed the need to support the hack for hardRef, along with the 
> management of deferred initialization of types.
>
> Let's merge this code in, given that it's a substantial improvement 
> over the old implementation, with all tests passing. We should 
> consider releasing a 2.7.2 soon, since this and related changes has 
> fixed serious problems in re and other mixed usage of Java and Python 
> via callbacks (http://bugs.jython.org/issue2609). Getting this out 
> would then allow us to work on Java 9 support in 2.7.3, as well as 
> fixing up sys slowness and related jsr223 (hopefully!).
>
> - Jim
>
>
>
> On Sun, Sep 24, 2017 at 3:19 AM, Jeff Allen <ja.py@... 
> <mailto:ja.py@...>> wrote:
>
>     As I suspected, the critical distinction is between the
>     *initiating* class loader and the *defining* class loader.
>
>     The defining class loader here turns out to be an instance of
>     sun.misc.Launcher$AppClassLoader, the grandparent of our
>     URLClassLoader that gets consulted before it tries the JAR URL we
>     gave. org.python.tests.Callbacker appears in jython-dev.jar, so it
>     is found there first. This is the same loader that holds PyObject,
>     etc., so essentially permanent and keeps our test class alive.
>
>     When this is all fixed by making the parent explicitly None in
>     make_jar_classloader, everything is GC'd as we'd hope. (Phew!)
>
>     https://bitbucket.org/tournesol/jython-fgtype/commits/4fd2baafb7499929f876ac20a607511668b3e222
>     <https://bitbucket.org/tournesol/jython-fgtype/commits/4fd2baafb7499929f876ac20a607511668b3e222>;
>
>     Jeff
>
>
>
>     On 18/09/2017 00:45, Jim Baker wrote:
>>     Jeff,
>>
>>     I looked at the new test, and it's interesting to see the
>>     ClassLoader gets collected, but not the corresponding PyJavaType.
>>     To me this implies the Class was necessarily collected; but it's
>>     not observable directly in Python code. While you are busy on
>>     other things, I will try playing with this changeset some more to
>>     see what is going on here. Maybe it ends just being a simple
>>     question of setting up a ref queue on Class GC and calling
>>     ClassValue#remove.
>>
>>     It's rather awesome about the simplification of fromClass, and
>>     removing the separate implementation of fromClassSkippingInners.
>>
>>     - Jim
>>
>>     On Sun, Sep 17, 2017 at 2:41 AM, Jeff Allen <ja.py@...
>>     <mailto:ja.py@...>> wrote:
>>
>>         Jim:
>>
>>         I figured out needsInners (I think!) and how I could make
>>         fromClass and fromClassSkippingInners into one method using
>>         the state of the registry to steer the action to be almost
>>         the same -- and reach the same conclusion -- with a net
>>         reduction in code.
>>
>>         https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e
>>         <https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e>;
>>
>>         I have added a test to prove the PyType (actually PyJavaType)
>>         is GC'd when we cease to reference it, using
>>         make_jar_classloader, but it is presently failing. Applied
>>         retrospectively to successive change sets it begins to fail
>>         exactly where I introduce ClassValue. :(
>>
>>         Maybe ClassValue doesn't quite work as I thought. Or I'm
>>         still not using it as John Rose intended. It seems wrong we
>>         should have to layer weak references on top or have some kind
>>         of removal queue.
>>
>>         I tried visualvm to see where it might be referenced, but I'm
>>         not sure how to interpret what it's telling me. However, this
>>         week, and maybe longer, other things have priority. Ideas
>>         welcome for when I'm back to this.
>>
>>         For exposed PyObjects, I'm pretty sure classToBuilder always
>>         was keeping a strong reference that would keep them alive
>>         now. (But not for the type under test.) I've removed this
>>         effect. But maybe we don't mind that?
>>         https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e#Lsrc/org/python/core/PyType.javaT1208
>>         <https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e#Lsrc/org/python/core/PyType.javaT1208>;
>>
>>         Jeff
>>
>>
>>         On 12/09/2017 01:37, Jim Baker wrote:
>>>         [I'm been having some problem posting to the jython-dev
>>>         mailing list, let's see if this response gets through!]
>>>
>>>         On Mon, Sep 11, 2017 at 1:47 PM, Jeff Allen
>>>         <ja.py@... <mailto:ja.py@...>> wrote:
>>>
>>>             Jim:
>>>
>>>             Thanks for the pointers. I agree with you on #2 that
>>>             what I've got *should* behave correctly with class
>>>             loaders, but is there a test you can suggest (or one
>>>             I've missed) that conclusively demonstrates it?
>>>
>>>
>>>         So I would start with test_support.make_jar_classloader,
>>>         just because it's easier to test this behavior in Python.
>>>         test_classpathimporter seems to use this support
>>>         functionality nicely.
>>>
>>>         If we want to add testing of our expectations around
>>>         ClassValue's lifetime, we can also add some sort of weakref
>>>         scheme, in conjunction with a multi-round GC, to verify that
>>>         the PyType gets collected as expected. Of course we should
>>>         expect ClassValue to behave as expected, but this would
>>>         ensure we are not holding anything, now or in the future
>>>         implementation, in the Jython runtime that would violate
>>>         this contract.
>>>
>>>
>>>             I'll press in on the question of inners. I tried just
>>>             calling fromClass, but it fails when the inner extends
>>>             the outer (test_import_jy:: test_selfreferential_classes
>>>             fails). You have to initialise the outer, then visit the
>>>             inners. In a way, it's the PyJavaType counterpart of the
>>>             deferred initialisation of exposed types that we control
>>>             by identifying bootstrap types. But here it has to be
>>>             self-guiding.
>>>
>>>             I have an idea that if I can make fromClass and
>>>             fromClassSkippingInners the same thing, then that can
>>>             become the implementation classToValue.computeValue, and
>>>             I won't need the NotReady exception or findExistingType.
>>>
>>>
>>>         So Java classes can form a general type graph (directed,
>>>         with cycles). It seems to me that the NotReady acts as some
>>>         sort of coloring in the graph exploration that fromClass is
>>>         doing. But as I said, I find this code for inner classes
>>>         rather subtle! Good luck in your coding here, it would be
>>>         really nice to get some further simplification.
>>>
>>>
>>>             Jeff
>>>
>>>             On 10/09/2017 01:12, Jim Baker wrote:
>>>
>>>                 Thanks for working on this change set, it's really
>>>                 good stuff. In particular, that's a very nice
>>>                 handoff you do in the PyType.Registry class,
>>>                 specifically moving computed types from
>>>                 classToNewType to classToType; and managing the
>>>                 NotReady state. Perhaps it needs some sort of state
>>>                 diagram to illustrate, but as someone who has been
>>>                 in that code, it makes perfect sense.
>>>
>>>                 With respect to GC, the following should be the case:
>>>
>>>                   * As long as the ClassLoader for the mapped Java
>>>                 class instance C is
>>>                     around, the corresponding PyType instance T will
>>>                 live. This
>>>                     follows from the definition of ClassValue as you
>>>                 point out in the
>>>                     code comments. (Good!)
>>>                   * As long as Python code holds a hard ref to T,
>>>                 the corresponding C
>>>                     will live. (Good!) (So we would see this in from
>>>                 `some_java_pkg
>>>                     import baz`; and it follows from the
>>>                 underlying_class field
>>>                     holding a hard ref from T to C.)
>>>                   * As long as Python code holds a hard ref to an
>>>                 instance of C, the
>>>                     corresponding T will also live. (Good!)
>>>
>>>                 Otherwise, this mapping will allow for GC of T, and
>>>                 subsequently of C, per the definition of ClassValue.
>>>                 I was also looking at Groovy's usage of a similar
>>>                 mapping, and corresponding bug,
>>>                 https://bugs.openjdk.java.net/browse/JDK-8136353
>>>                 <https://bugs.openjdk.java.net/browse/JDK-8136353>;,
>>>                 and it looks like the problem reported there doesn't
>>>                 hold. This is because there are no strong refs
>>>                 outside of PyType itself.
>>>
>>>                 So everything looks good for getting this change in.
>>>
>>>                 On Sat, Sep 9, 2017 at 2:12 PM, Jeff Allen
>>>                 <ja.py@... <mailto:ja.py@...>
>>>                 <mailto:ja.py@...
>>>                 <mailto:ja.py@...>>> wrote:
>>>
>>>                     Jim, all:
>>>
>>>                     By a series of small steps I've come quite a
>>>                 long way in this
>>>                     aspect of PyType. It might be worth a look.
>>>                 https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd
>>>                 <https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd>;
>>>                    
>>>                 <https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd
>>>                 <https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd>>;
>>>
>>>                     In this last commit I've entirely dispensed with
>>>                 any long-term
>>>                     reference to the created PyType in the type
>>>                 registry: only the
>>>                     ClassValue holds one. The explicit management of
>>>                 reachability goes
>>>                     away and we have just the behaviour given us by
>>>                 ClassValue. The
>>>                     only down-side I see is that a PyType created
>>>                 for a Java class you
>>>                     handle briefly from Python will not be
>>>                 collectable until the class is.
>>>
>>>                     The old test for weakness, which I've been
>>>                 running in parallel
>>>                     with the new, requires PyType.getClassToType(),
>>>                 and can no longer
>>>                     be supported. (You can't enumerate the
>>>                 ClassValue.) I believe the
>>>                     replacement tests the same thing -- they used to
>>>                 pass and fail
>>>                     together.
>>>
>>>
>>>                 Right, it makes sense to remove this test. This is
>>>                 because we don't want to support the necessary
>>>                 mapping, even though it could be readily supported
>>>                 with an appropriate concurrent weak mapping - the
>>>                 only semantics it would be supporting is for this
>>>                 very test.
>>>
>>>
>>>                     I've moved some actions around, and removed some
>>>                 of the checks. We
>>>                     seemingly expected the desired PyType to be
>>>                 created at any moment
>>>                     by a side effect. Reason and regression testing
>>>                 tell me that
>>>                     doesn't happen in the new arrangement. I think
>>>                 there is slightly
>>>                     less code to execute, but masses more comment
>>>                 and debug -- it's my
>>>                     way of figuring it out. It's all very educational.
>>>
>>>                     I still don't understand:
>>>
>>>                     1. Whether we find the downside (PyType not GC'd
>>>                 until the class is)
>>>                        important.
>>>
>>>
>>>                 So we do know that Jython will continue to work if
>>>                 we remove a mapping entry, since it happens all the
>>>                 time currently in some code like json, re, and
>>>                 socket, where classes are not directly imported :)
>>>                 We could always do some sort of LRU and then call
>>>                 ClassValue#remove, but I don't think it makes sense.
>>>                 There are many other other optimizations in the
>>>                 runtime first.
>>>
>>>                     2. Whether I have broken something that the
>>>                 regression tests don't
>>>                        cover (like use with isolating class loaders).
>>>
>>>
>>>                 These are different Java classes if in different
>>>                 class loaders. For Python usage of these Java
>>>                 classes, it's all the same, thanks to duck typing,
>>>                 but we need to maintain a different mapping entry.
>>>                 Again #1 demonstrates we have been handling this
>>>                 correctly in the current codebase.
>>>
>>>                     3. What needsInners is really up to. I see
>>>                 *what* it does, but not
>>>                        *why* well enough to consider alternatives.
>>>                 It would simplify
>>>                        computeValue if there were no
>>>                 fromClassSkippingInners: could
>>>                        needsInners be a registry global? And why do
>>>                 we skip processing
>>>                        them, when we do?
>>>
>>>
>>>                 That codepath is exceedingly subtle, at least to me.
>>>                 But it would be an interesting experiment to see if
>>>                 it could be made global, vs this potentially
>>>                 expanding context that is passed through.
>>>
>>>                 - Jim
>>>
>>>
>>>
>>
>>
>
>

Re: [Jython-dev] PyType publication with better concurrency

[Jim Baker <jim.baker@py...>]

Likewise, glad to see it was just a problem in the test setup. Everything
looks good now: the test demonstrates that the Python type mapping does not
prevent class GC, assuming nothing else refers to the Java class. Indeed
the earlier faulty version of the test further demonstrates that point,
given the parent classloader referring to the class, and preventing GC.

I have also had a chance to review the rest of the change set, and
everything else looks great. In particular, I like how you have removed the
need to support the hack for hardRef, along with the management of deferred
initialization of types.

Let's merge this code in, given that it's a substantial improvement over
the old implementation, with all tests passing. We should consider
releasing a 2.7.2 soon, since this and related changes has fixed serious
problems in re and other mixed usage of Java and Python via callbacks (
http://bugs.jython.org/issue2609). Getting this out would then allow us to
work on Java 9 support in 2.7.3, as well as fixing up sys slowness and
related jsr223 (hopefully!).

- Jim



On Sun, Sep 24, 2017 at 3:19 AM, Jeff Allen <ja.py@...> wrote:

> As I suspected, the critical distinction is between the *initiating*
> class loader and the *defining* class loader.
>
> The defining class loader here turns out to be an instance of
> sun.misc.Launcher$AppClassLoader, the grandparent of our URLClassLoader
> that gets consulted before it tries the JAR URL we gave.
> org.python.tests.Callbacker appears in jython-dev.jar, so it is found there
> first. This is the same loader that holds PyObject, etc., so essentially
> permanent and keeps our test class alive.
>
> When this is all fixed by making the parent explicitly None in
> make_jar_classloader, everything is GC'd as we'd hope. (Phew!)
>
> https://bitbucket.org/tournesol/jython-fgtype/commits/
> 4fd2baafb7499929f876ac20a607511668b3e222
> Jeff
>
>
>
> On 18/09/2017 00:45, Jim Baker wrote:
>
> Jeff,
>
> I looked at the new test, and it's interesting to see the ClassLoader gets
> collected, but not the corresponding PyJavaType. To me this implies the
> Class was necessarily collected; but it's not observable directly in Python
> code. While you are busy on other things, I will try playing with this
> changeset some more to see what is going on here. Maybe it ends just being
> a simple question of setting up a ref queue on Class GC and calling
> ClassValue#remove.
>
> It's rather awesome about the simplification of fromClass, and removing
> the separate implementation of fromClassSkippingInners.
>
> - Jim
>
> On Sun, Sep 17, 2017 at 2:41 AM, Jeff Allen <ja.py@...> wrote:
>
>> Jim:
>>
>> I figured out needsInners (I think!) and how I could make fromClass and
>> fromClassSkippingInners into one method using the state of the registry to
>> steer the action to be almost the same -- and reach the same conclusion --
>> with a net reduction in code.
>>
>> https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc
>> 1f52c9b3733841b375831f8a3c0d27a93e
>>
>> I have added a test to prove the PyType (actually PyJavaType) is GC'd
>> when we cease to reference it, using make_jar_classloader, but it is
>> presently failing. Applied retrospectively to successive change sets it
>> begins to fail exactly where I introduce ClassValue. :(
>> Maybe ClassValue doesn't quite work as I thought. Or I'm still not using
>> it as John Rose intended. It seems wrong we should have to layer weak
>> references on top or have some kind of removal queue.
>>
>> I tried visualvm to see where it might be referenced, but I'm not sure
>> how to interpret what it's telling me. However, this week, and maybe
>> longer, other things have priority. Ideas welcome for when I'm back to this.
>> For exposed PyObjects, I'm pretty sure classToBuilder always was keeping
>> a strong reference that would keep them alive now. (But not for the type
>> under test.) I've removed this effect. But maybe we don't mind that?
>> https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc
>> 1f52c9b3733841b375831f8a3c0d27a93e#Lsrc/org/python/core/PyType.javaT1208
>>
>> Jeff
>>
>>
>> On 12/09/2017 01:37, Jim Baker wrote:
>>
>> [I'm been having some problem posting to the jython-dev mailing list,
>> let's see if this response gets through!]
>>
>> On Mon, Sep 11, 2017 at 1:47 PM, Jeff Allen <ja.py@...> wrote:
>>
>>> Jim:
>>>
>>> Thanks for the pointers. I agree with you on #2 that what I've got
>>> *should* behave correctly with class loaders, but is there a test you can
>>> suggest (or one I've missed) that conclusively demonstrates it?
>>>
>>
>> So I would start with test_support.make_jar_classloader, just because
>> it's easier to test this behavior in Python. test_classpathimporter seems
>> to use this support functionality nicely.
>>
>> If we want to add testing of our expectations around ClassValue's
>> lifetime, we can also add some sort of weakref scheme, in conjunction with
>> a multi-round GC, to verify that the PyType gets collected as expected. Of
>> course we should expect ClassValue to behave as expected, but this would
>> ensure we are not holding anything, now or in the future implementation, in
>> the Jython runtime that would violate this contract.
>>
>>
>>> I'll press in on the question of inners. I tried just calling fromClass,
>>> but it fails when the inner extends the outer (test_import_jy::
>>> test_selfreferential_classes fails). You have to initialise the outer, then
>>> visit the inners. In a way, it's the PyJavaType counterpart of the deferred
>>> initialisation of exposed types that we control by identifying bootstrap
>>> types. But here it has to be self-guiding.
>>>
>>> I have an idea that if I can make fromClass and fromClassSkippingInners
>>> the same thing, then that can become the implementation
>>> classToValue.computeValue, and I won't need the NotReady exception or
>>> findExistingType.
>>>
>>
>> So Java classes can form a general type graph (directed, with cycles). It
>> seems to me that the NotReady acts as some sort of coloring in the graph
>> exploration that fromClass is doing. But as I said, I find this code for
>> inner classes rather subtle! Good luck in your coding here, it would be
>> really nice to get some further simplification.
>>
>>
>>>
>>> Jeff
>>>
>>> On 10/09/2017 01:12, Jim Baker wrote:
>>>
>>>> Thanks for working on this change set, it's really good stuff. In
>>>> particular, that's a very nice handoff you do in the PyType.Registry class,
>>>> specifically moving computed types from classToNewType to classToType; and
>>>> managing the NotReady state. Perhaps it needs some sort of state diagram to
>>>> illustrate, but as someone who has been in that code, it makes perfect
>>>> sense.
>>>>
>>>> With respect to GC, the following should be the case:
>>>>
>>>>   * As long as the ClassLoader for the mapped Java class instance C is
>>>>     around, the corresponding PyType instance T will live. This
>>>>     follows from the definition of ClassValue as you point out in the
>>>>     code comments. (Good!)
>>>>   * As long as Python code holds a hard ref to T, the corresponding C
>>>>     will live. (Good!) (So we would see this in from `some_java_pkg
>>>>     import baz`; and it follows from the underlying_class field
>>>>     holding a hard ref from T to C.)
>>>>   * As long as Python code holds a hard ref to an instance of C, the
>>>>     corresponding T will also live. (Good!)
>>>>
>>>> Otherwise, this mapping will allow for GC of T, and subsequently of C,
>>>> per the definition of ClassValue. I was also looking at Groovy's usage of a
>>>> similar mapping, and corresponding bug, https://bugs.openjdk.java.net/
>>>> browse/JDK-8136353, and it looks like the problem reported there
>>>> doesn't hold. This is because there are no strong refs outside of PyType
>>>> itself.
>>>>
>>>> So everything looks good for getting this change in.
>>>>
>>>> On Sat, Sep 9, 2017 at 2:12 PM, Jeff Allen <ja.py@... <mailto:
>>>> ja.py@...>> wrote:
>>>>
>>>>     Jim, all:
>>>>
>>>>     By a series of small steps I've come quite a long way in this
>>>>     aspect of PyType. It might be worth a look.
>>>>     https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8
>>>> ff48bf5d20cb57169cd3f43a3cd90df0cd
>>>>     <https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d
>>>> 8ff48bf5d20cb57169cd3f43a3cd90df0cd>
>>>>
>>>>     In this last commit I've entirely dispensed with any long-term
>>>>     reference to the created PyType in the type registry: only the
>>>>     ClassValue holds one. The explicit management of reachability goes
>>>>     away and we have just the behaviour given us by ClassValue. The
>>>>     only down-side I see is that a PyType created for a Java class you
>>>>     handle briefly from Python will not be collectable until the class
>>>> is.
>>>>
>>>>     The old test for weakness, which I've been running in parallel
>>>>     with the new, requires PyType.getClassToType(), and can no longer
>>>>     be supported. (You can't enumerate the ClassValue.) I believe the
>>>>     replacement tests the same thing -- they used to pass and fail
>>>>     together.
>>>>
>>>>
>>>> Right, it makes sense to remove this test. This is because we don't
>>>> want to support the necessary mapping, even though it could be readily
>>>> supported with an appropriate concurrent weak mapping - the only semantics
>>>> it would be supporting is for this very test.
>>>>
>>>>
>>>>     I've moved some actions around, and removed some of the checks. We
>>>>     seemingly expected the desired PyType to be created at any moment
>>>>     by a side effect. Reason and regression testing tell me that
>>>>     doesn't happen in the new arrangement. I think there is slightly
>>>>     less code to execute, but masses more comment and debug -- it's my
>>>>     way of figuring it out. It's all very educational.
>>>>
>>>>     I still don't understand:
>>>>
>>>>     1. Whether we find the downside (PyType not GC'd until the class is)
>>>>        important.
>>>>
>>>>
>>>> So we do know that Jython will continue to work if we remove a mapping
>>>> entry, since it happens all the time currently in some code like json, re,
>>>> and socket, where classes are not directly imported :) We could always do
>>>> some sort of LRU and then call ClassValue#remove, but I don't think it
>>>> makes sense. There are many other other optimizations in the runtime first.
>>>>
>>>>     2. Whether I have broken something that the regression tests don't
>>>>        cover (like use with isolating class loaders).
>>>>
>>>>
>>>> These are different Java classes if in different class loaders. For
>>>> Python usage of these Java classes, it's all the same, thanks to duck
>>>> typing, but we need to maintain a different mapping entry. Again #1
>>>> demonstrates we have been handling this correctly in the current codebase.
>>>>
>>>>     3. What needsInners is really up to. I see *what* it does, but not
>>>>        *why* well enough to consider alternatives. It would simplify
>>>>        computeValue if there were no fromClassSkippingInners: could
>>>>        needsInners be a registry global? And why do we skip processing
>>>>        them, when we do?
>>>>
>>>>
>>>> That codepath is exceedingly subtle, at least to me. But it would be an
>>>> interesting experiment to see if it could be made global, vs this
>>>> potentially expanding context that is passed through.
>>>>
>>>> - Jim
>>>>
>>>
>>>
>>
>>
>
>

Re: [Jython-dev] PyType publication with better concurrency

[Jeff Allen <ja.py@fa...>]

As I suspected, the critical distinction is between the *initiating* 
class loader and the *defining* class loader.

The defining class loader here turns out to be an instance of 
sun.misc.Launcher$AppClassLoader, the grandparent of our URLClassLoader 
that gets consulted before it tries the JAR URL we gave. 
org.python.tests.Callbacker appears in jython-dev.jar, so it is found 
there first. This is the same loader that holds PyObject, etc., so 
essentially permanent and keeps our test class alive.

When this is all fixed by making the parent explicitly None in 
make_jar_classloader, everything is GC'd as we'd hope. (Phew!)

https://bitbucket.org/tournesol/jython-fgtype/commits/4fd2baafb7499929f876ac20a607511668b3e222

Jeff


On 18/09/2017 00:45, Jim Baker wrote:
> Jeff,
>
> I looked at the new test, and it's interesting to see the ClassLoader 
> gets collected, but not the corresponding PyJavaType. To me this 
> implies the Class was necessarily collected; but it's not observable 
> directly in Python code. While you are busy on other things, I will 
> try playing with this changeset some more to see what is going on 
> here. Maybe it ends just being a simple question of setting up a ref 
> queue on Class GC and calling ClassValue#remove.
>
> It's rather awesome about the simplification of fromClass, and 
> removing the separate implementation of fromClassSkippingInners.
>
> - Jim
>
> On Sun, Sep 17, 2017 at 2:41 AM, Jeff Allen <ja.py@... 
> <mailto:ja.py@...>> wrote:
>
>     Jim:
>
>     I figured out needsInners (I think!) and how I could make
>     fromClass and fromClassSkippingInners into one method using the
>     state of the registry to steer the action to be almost the same --
>     and reach the same conclusion -- with a net reduction in code.
>
>     https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e
>     <https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e>;
>
>     I have added a test to prove the PyType (actually PyJavaType) is
>     GC'd when we cease to reference it, using make_jar_classloader,
>     but it is presently failing. Applied retrospectively to successive
>     change sets it begins to fail exactly where I introduce ClassValue. :(
>
>     Maybe ClassValue doesn't quite work as I thought. Or I'm still not
>     using it as John Rose intended. It seems wrong we should have to
>     layer weak references on top or have some kind of removal queue.
>
>     I tried visualvm to see where it might be referenced, but I'm not
>     sure how to interpret what it's telling me. However, this week,
>     and maybe longer, other things have priority. Ideas welcome for
>     when I'm back to this.
>
>     For exposed PyObjects, I'm pretty sure classToBuilder always was
>     keeping a strong reference that would keep them alive now. (But
>     not for the type under test.) I've removed this effect. But maybe
>     we don't mind that?
>     https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e#Lsrc/org/python/core/PyType.javaT1208
>     <https://bitbucket.org/tournesol/jython-fgtype/commits/f781dc1f52c9b3733841b375831f8a3c0d27a93e#Lsrc/org/python/core/PyType.javaT1208>;
>
>     Jeff
>
>
>     On 12/09/2017 01:37, Jim Baker wrote:
>>     [I'm been having some problem posting to the jython-dev mailing
>>     list, let's see if this response gets through!]
>>
>>     On Mon, Sep 11, 2017 at 1:47 PM, Jeff Allen <ja.py@...
>>     <mailto:ja.py@...>> wrote:
>>
>>         Jim:
>>
>>         Thanks for the pointers. I agree with you on #2 that what
>>         I've got *should* behave correctly with class loaders, but is
>>         there a test you can suggest (or one I've missed) that
>>         conclusively demonstrates it?
>>
>>
>>     So I would start with test_support.make_jar_classloader, just
>>     because it's easier to test this behavior in Python.
>>     test_classpathimporter seems to use this support functionality
>>     nicely.
>>
>>     If we want to add testing of our expectations around ClassValue's
>>     lifetime, we can also add some sort of weakref scheme, in
>>     conjunction with a multi-round GC, to verify that the PyType gets
>>     collected as expected. Of course we should expect ClassValue to
>>     behave as expected, but this would ensure we are not holding
>>     anything, now or in the future implementation, in the Jython
>>     runtime that would violate this contract.
>>
>>
>>         I'll press in on the question of inners. I tried just calling
>>         fromClass, but it fails when the inner extends the outer
>>         (test_import_jy:: test_selfreferential_classes fails). You
>>         have to initialise the outer, then visit the inners. In a
>>         way, it's the PyJavaType counterpart of the deferred
>>         initialisation of exposed types that we control by
>>         identifying bootstrap types. But here it has to be self-guiding.
>>
>>         I have an idea that if I can make fromClass and
>>         fromClassSkippingInners the same thing, then that can become
>>         the implementation classToValue.computeValue, and I won't
>>         need the NotReady exception or findExistingType.
>>
>>
>>     So Java classes can form a general type graph (directed, with
>>     cycles). It seems to me that the NotReady acts as some sort of
>>     coloring in the graph exploration that fromClass is doing. But as
>>     I said, I find this code for inner classes rather subtle! Good
>>     luck in your coding here, it would be really nice to get some
>>     further simplification.
>>
>>
>>         Jeff
>>
>>         On 10/09/2017 01:12, Jim Baker wrote:
>>
>>             Thanks for working on this change set, it's really good
>>             stuff. In particular, that's a very nice handoff you do
>>             in the PyType.Registry class, specifically moving
>>             computed types from classToNewType to classToType; and
>>             managing the NotReady state. Perhaps it needs some sort
>>             of state diagram to illustrate, but as someone who has
>>             been in that code, it makes perfect sense.
>>
>>             With respect to GC, the following should be the case:
>>
>>               * As long as the ClassLoader for the mapped Java class
>>             instance C is
>>                 around, the corresponding PyType instance T will
>>             live. This
>>                 follows from the definition of ClassValue as you
>>             point out in the
>>                 code comments. (Good!)
>>               * As long as Python code holds a hard ref to T, the
>>             corresponding C
>>                 will live. (Good!) (So we would see this in from
>>             `some_java_pkg
>>                 import baz`; and it follows from the underlying_class
>>             field
>>                 holding a hard ref from T to C.)
>>               * As long as Python code holds a hard ref to an
>>             instance of C, the
>>                 corresponding T will also live. (Good!)
>>
>>             Otherwise, this mapping will allow for GC of T, and
>>             subsequently of C, per the definition of ClassValue. I
>>             was also looking at Groovy's usage of a similar mapping,
>>             and corresponding bug,
>>             https://bugs.openjdk.java.net/browse/JDK-8136353
>>             <https://bugs.openjdk.java.net/browse/JDK-8136353>;, and
>>             it looks like the problem reported there doesn't hold.
>>             This is because there are no strong refs outside of
>>             PyType itself.
>>
>>             So everything looks good for getting this change in.
>>
>>             On Sat, Sep 9, 2017 at 2:12 PM, Jeff Allen
>>             <ja.py@... <mailto:ja.py@...>
>>             <mailto:ja.py@... <mailto:ja.py@...>>>
>>             wrote:
>>
>>                 Jim, all:
>>
>>                 By a series of small steps I've come quite a long way
>>             in this
>>                 aspect of PyType. It might be worth a look.
>>             https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd
>>             <https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd>;
>>                
>>             <https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd
>>             <https://bitbucket.org/tournesol/jython-fgtype/commits/3fc5d8ff48bf5d20cb57169cd3f43a3cd90df0cd>>;
>>
>>                 In this last commit I've entirely dispensed with any
>>             long-term
>>                 reference to the created PyType in the type registry:
>>             only the
>>                 ClassValue holds one. The explicit management of
>>             reachability goes
>>                 away and we have just the behaviour given us by
>>             ClassValue. The
>>                 only down-side I see is that a PyType created for a
>>             Java class you
>>                 handle briefly from Python will not be collectable
>>             until the class is.
>>
>>                 The old test for weakness, which I've been running in
>>             parallel
>>                 with the new, requires PyType.getClassToType(), and
>>             can no longer
>>                 be supported. (You can't enumerate the ClassValue.) I
>>             believe the
>>                 replacement tests the same thing -- they used to pass
>>             and fail
>>                 together.
>>
>>
>>             Right, it makes sense to remove this test. This is
>>             because we don't want to support the necessary mapping,
>>             even though it could be readily supported with an
>>             appropriate concurrent weak mapping - the only semantics
>>             it would be supporting is for this very test.
>>
>>
>>                 I've moved some actions around, and removed some of
>>             the checks. We
>>                 seemingly expected the desired PyType to be created
>>             at any moment
>>                 by a side effect. Reason and regression testing tell
>>             me that
>>                 doesn't happen in the new arrangement. I think there
>>             is slightly
>>                 less code to execute, but masses more comment and
>>             debug -- it's my
>>                 way of figuring it out. It's all very educational.
>>
>>                 I still don't understand:
>>
>>                 1. Whether we find the downside (PyType not GC'd
>>             until the class is)
>>                    important.
>>
>>
>>             So we do know that Jython will continue to work if we
>>             remove a mapping entry, since it happens all the time
>>             currently in some code like json, re, and socket, where
>>             classes are not directly imported :) We could always do
>>             some sort of LRU and then call ClassValue#remove, but I
>>             don't think it makes sense. There are many other other
>>             optimizations in the runtime first.
>>
>>                 2. Whether I have broken something that the
>>             regression tests don't
>>                    cover (like use with isolating class loaders).
>>
>>
>>             These are different Java classes if in different class
>>             loaders. For Python usage of these Java classes, it's all
>>             the same, thanks to duck typing, but we need to maintain
>>             a different mapping entry. Again #1 demonstrates we have
>>             been handling this correctly in the current codebase.
>>
>>                 3. What needsInners is really up to. I see *what* it
>>             does, but not
>>                    *why* well enough to consider alternatives. It
>>             would simplify
>>                    computeValue if there were no
>>             fromClassSkippingInners: could
>>                    needsInners be a registry global? And why do we
>>             skip processing
>>                    them, when we do?
>>
>>
>>             That codepath is exceedingly subtle, at least to me. But
>>             it would be an interesting experiment to see if it could
>>             be made global, vs this potentially expanding context
>>             that is passed through.
>>
>>             - Jim
>>
>>
>>
>
>

[Jython-dev] JyNI 2.7-alpha.5 released: JyNI also runs on Windows now

[Stefan Richthofer <Stefan.Richthofer@gm...>]

With JyNI alpha 5, I proudly announce the first version of JyNI that
also runs on Windows.

JyNI is a layer that allows Jython to load native CPython extensions.
Most notably ctypes and NumPy are already workable (core functionality).


Take a look at

jyni.org

or go directly to the release:

https://github.com/Stewori/JyNI/releases/tag/v2.7-alpha.5


Binaries are provided for Linux, OS X and Windows.
It should in principle also work on other POSIX platforms.



NumPy support
-------------

>From JyNI alpha 4 onwards, some NumPy support is available.
For JyNI alpha 5 featuring Windows support, we explicitly asserted that
NumPy works on Windows equally well as on Linux and OS X,
i.e. up to known limitations.

For details about NumPy support see

https://github.com/Stewori/JyNI/releases/tag/v2.7-alpha.4
https://github.com/Stewori/JyNI/issues/2



Google Summer of Code
---------------------

JyNI alpha 5 was mainly developed within a Google Summer of Code project:

https://summerofcode.withgoogle.com/projects/#4931062149939200
http://gsoc2017-jyni.blogspot.de for details.



What comes next?
----------------

The Python C API is not yet fully supported. A major goal for the next
release is to add support for the buffer protocol, which is currently
blocking support for SciPy and various other extensions.



Enjoy!


-Stefan

[Jython-dev] Summary of Jython tracker Issues

[Jython tracker <status@bu...>]

ACTIVITY SUMMARY (2017-09-15 - 2017-09-22)
Jython tracker at http://bugs.jython.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open     317 ( +0)
  closed  2327 ( +0)
  total   2644 ( +0)

Open issues with patches: 28   


Most recent 15 issues with no replies (15)
==========================================

#2618: socket.sendall no longer sends all
http://bugs.jython.org/issue2618

#2616: Incomplete / broken support for Certificate Revocation Lists
http://bugs.jython.org/issue2616

#2613: 2.7.1 MANIFEST.MF contains unfilled placeholders
http://bugs.jython.org/issue2613

#2611: mkdir() operation in /Lib/os.py has different behavior when ru
http://bugs.jython.org/issue2611

#2606: jython launch problem sun.misc.InvalidJarIndexException
http://bugs.jython.org/issue2606

#2605: Jython 2.7.0 startup performance regression
http://bugs.jython.org/issue2605

#2591: Unable to execute directory or zip file (test_cmd_line_script)
http://bugs.jython.org/issue2591

#2567: System state lost during JSR-223 initialisation
http://bugs.jython.org/issue2567

#2562: Windows: OSError: unlink(): an unknown error occurred
http://bugs.jython.org/issue2562

#2531: Support SNI for SSL/TLS server sockets
http://bugs.jython.org/issue2531

#2525: Jython incorrectly buffers file pipe output with Subprocess(co
http://bugs.jython.org/issue2525

#2520: Jython does NOT support socket.listen(0) for accepting only on
http://bugs.jython.org/issue2520

#2512: Values in built-in modules's __dict__ are “<reflected field 
http://bugs.jython.org/issue2512

#2507: Run "invokeFunction" in a thread does not inherit the "ScriptC
http://bugs.jython.org/issue2507

#2494: Support for pydoc_data
http://bugs.jython.org/issue2494



Most recent 15 issues waiting for review (15)
=============================================

#2566: inspect does not recognize code objects from bytecode files
http://bugs.jython.org/issue2566

#2545: help() does not work on Java-implemented modules
http://bugs.jython.org/issue2545

#2429: cStringIO does not work with mutable objects implementing the 
http://bugs.jython.org/issue2429

#2367: Jython ignores custom __eq__ when hashing dict subclasses
http://bugs.jython.org/issue2367

#2330: full-build fails to copy CPython License
http://bugs.jython.org/issue2330

#2230: Jython evaluation blocks under heavy load with high multi-core
http://bugs.jython.org/issue2230

#2143: site-packages support in standalone jar
http://bugs.jython.org/issue2143

#2142: Set Thread classloader when entering Jython context
http://bugs.jython.org/issue2142

#2121: Jython jar on Maven central embeds other third party libraries
http://bugs.jython.org/issue2121

#2077: marshal doesn't raise error when fed unmarshalable object
http://bugs.jython.org/issue2077

#1925: Support loading java.sql.Drivers that aren't on the boot class
http://bugs.jython.org/issue1925

#1917: No ctypes.c_char
http://bugs.jython.org/issue1917

#1866: Parser does not have mismatch token error messages caught by B
http://bugs.jython.org/issue1866

#1842: Add IBM i support to Jython
http://bugs.jython.org/issue1842

#1796: Jython doesn't support jar dir with colon's in it
http://bugs.jython.org/issue1796

[Jython-dev] Release Announcement: General Availability of JDK 9

[Rory O'Donnell <rory.odonnell@or...>]

Hi Alan,

Three items to share with you today

  * *JDK 9 General Availability
    *
      o GPL'd binaries from Oracle are available here:
          + http://jdk.java.net/9
      o See Mark Reinhold's email for more details on the Release [1]
          + delivery of Project Jigsaw [2]

  * Are you JDK 9 Ready ?
      o The Quality Outreach wiki has been updated to include a JDK 9
        Ready column.
      o If you would like us to identify your project as JDK 9 ready ,
        please let me know and I will add it to the wiki.

  * Quality Outreach Report for September 2017**is available
      o many thanks for your continued support and welcome to the new
        projects!

Rgds,Rory

[1] 
http://mail.openjdk.java.net/pipermail/announce/2017-September/000230.html
[2] https://mreinhold.org/blog/jigsaw-complete

-- 
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland