During the past week, all of the bugs I suggested be fixed for beta 3 have been fixed, along with a number of others. With this work done, we are now ready to release a beta 3.

It was also really impressive seeing this work come together.

In particular, Indra Talip contributed a fix to binascii (a helper library for such functionality as base64) to have it work with bytearray and memoryview objects, as well as str objects. Indra did this by nicely using Jeff Allen's port of the BufferProtocol from CPython, which allows Jython to consistently and efficiently work with various byte layouts, whether bytearray (actually byte[]) or str (actually uses a java.lang.String as its backing, therefore each byte is actually sorted as a char for a byte, to more easily interoperate with Java).

Interestingly in CPython 2.7's implementation of binascii, a copy is made to str (PyString). Only in CPython 3 does it more to actually using Py_buffer (as least of 3.4). So here and a number of other places, we are actually doing advance work for Jython 3.x.

Indra also identified that we had not recently run gderived.py, so I updated all *Derived classes accordingly. This fixes http://bugs.jython.org/issue2119

Another aspect that made development more smooth for people contributing code is that the mirroring of hg.python.org/jython to the bitbucket repo at https://bitbucket.org/jython/jython was fixed by bitbucket this week. So we are now in the position of actually being able to work with pull requests again.

Lastly, I should mention the process for merging such pull requests, especially given this mirroring and lack of continuous integration for pull requests, while still having the PRs marked as merged by bitbucket. In a nutshell:

$ hg pull -r CHANGESET1 -r CHANGESET2 ... BRANCH

So for me to pull the bz2 fixes from Indra's branch, I looked at the corresponding commits in the PR about bz2 and constructed this command:

hg pull -r 1241914 -r b73f07d -r b1304c8 https://bitbucket.org/indratalip/jython/branch/default

The ordering of the changesets must be from oldest to newest. Next, I verified with hg log that the commits were in the same order, then ran the regrtest as usual. After completing the code review itself, I can push to hg.python.org/jython, then see it re-mirror at bitbucket.

It is not possible to rebase these changesets, because they are immutable.

A bit tedious, but this workflow allows us to maintain our code at hg.python.org, while still working with other tooling that's more user friendly for potential contributors. I have tried doing this other ways, such as committing the diff for a contributor, but this results in PRs that have no changes, but are not marked as merged, because there's no linking changeset. We will be "declining" such PRs to clean up the PR queue, then hopefully avoiding such problems in the future.

- Jim

On Wed, May 7, 2014 at 11:53 PM, Jim Baker <jim.baker@python.org> wrote:
The socket reboot work is finally merged in to trunk! So with Jython trunk, the following now works:
More below:

On Sun, May 4, 2014 at 4:01 PM, Jim Baker <jim.baker@python.org> wrote:
I expect that we will have a soft beta 3 available by the end of this week! This will mean Jython will regain support for pip and easy_install when installing packages from PyPI, which we lost when PyPA tools moved to requiring ssl. In addition, this work will add support for the Python wheel distribution format, the fantastic requests package, and many other fixes that can be seen in the commit log.

Here is my plan for the remaining work for beta 3; I would appreciate any feedback:

Merged in changeset 7222:107fe4a4c96b

This proved to be trickier than I expected because my socket branch itself included a number of branches, and hg.python.org would not accept them, but I finally found a way to collapse together all the changesets (edit history) into one and thereby work around this issue. This is probably a nicer way to see it anyway.

More here on the approach I took: http://mercurial.selenic.com/wiki/ConcatenatingChangesets - I chose to use the "revert solution".

Fortunately, I do not expect any other "reboots" of this complexity to be done for 2.7.0!

I have made a pull request against html5lib-python (https://github.com/html5lib/html5lib-python/pull/150) to upstream a fix I have made in pip in this vendor lib dependency (https://github.com/jimbaker/pip). For now, Jython users will need to use my pip branch to have pip work.

So this pull request is now on hold pending CJK codecs (http://bugs.jython.org/issue1066)

... I also have not tested socket reboot against Windows yet, so if anyone has some cycles to do that against my branch, I would appreciate it. I do have a MSDN license as a Python committer (thanks Microsoft!), but I'm currently somewhat time constrained. My expectation is that we can take advantage of the portability work that Netty has done.

I have done this testing against Windows 8.1, and except for one test in TestJython_get_jsockaddr, all tests in test_socket pass. We will have to look at refactoring this specific test suite, which also fails on one of my Linux boxes, due to the fact that it can depend on the specific networking configuration of the host.

There are some additional changes I would like to see make beta 3:

These are next. I still would like to have all such work completed by the end of this week however.

contributed by Indra Talip - Fix the struct module so that struct.Struct class can be derived from

contributed by Indra Talip - Fix os.stat behaviour for paths with trailing slash when the path exists as a file

contributed by Arfrever Frehtes Taifersar Arahesis - Fix compatibility with jnr-posix 3.0

Replace Guava MapMaker.makeComputingMap with CacheBuilder
This will allow us to upgrade Guava to release 17

defaultdict get behavior causes inconsistent map entries leading to potential memory leaks
Requires fixing #2087 and motivating this, since memory leaks are an obvious priority