One more thing on the AST being correct or not. We are using CPython's _ast module to verify the correctness of our own AST generation. If you look at the sandbox ast regr script, you will see that we test against CPython 2.5's library. If we have the same parse, we pass. So this is a good example of how we are using test-driven development.

Of course, it could be a lot better testing. (I even promised Frank I'd help there. But so far too little time to do everything that should be done!) But it is systematic, strong testing, and that's what counts.

- Jim

On Jan 31, 2008 8:00 PM, Jim Baker <> wrote:

On Jan 31, 2008 6:50 PM, Alexandru Popescu ☀ <> wrote:
On Feb 1, 2008 3:33 AM, Frank Wierzbicki <> wrote:
> On Jan 31, 2008 6:56 PM, Alexandru Popescu

> > 2/ there is a python bytecode to java bytecode converter (which
> > according to people handles correctly "all" conversions).
> > Currently pyasm is written in python and work on python bytecode directly.
> > This lives in another sandbox project: sandbox/pyasm.
> Right -- that is the work of Jim Baker and Tobias Ivarsson.
> > However, moving forward doesn't only mean to integrate these 2
> > subprojects into the trunk. There is still a missing step in the
> > middle of processing:
> > a) adapt pyasm to work on the parser generated AST
> > b) add a post parser step that takes as input the AST and generates
> > python bytecode. After this step pyasm can probably be applied as is
> > (or with very small changes).
> Well not python bytecode but JVM bytecode -- but maybe that is what
> you meant?  Also we need to translate pyasm to pure Java since we are
> not pypy (though who knows about the future...)

No, I actually meant what I have written down :-). Note, I'm speaking
about 2 possible approaches:

a/ (that is the most natural) move pyasm to java and apply it to the
AST instead of python bytecode => java bytecode

b/ (would be a twisted approach, but would probably offer a guarantee
that the AST is the most correct one)

AST => python bytecode (through a transformer) => java bytecode (through pyasm)

We are planning on supporting option a for now. But as Tobias observed to me, option b is actually quite useful seen through a different perspective, if you modify that final transformation by pyasm. So the idea would be to have Python bytecode VM that is available in Jython. PBC is quite high level, and we already know how to compile it to Java bytecode that works with the Jython internals using pyasm. So interpreting it should be really easy. The advantage of doing so: we could provide the missing level of dynamic functionality that would otherwise require runtime bytecode gen. So it could run on Dalvik or in an applet.

JRuby incidentally supports both interpreted and compiled mode, although I don't know if they have a similar VM approach.

I do believe that a/ is the right approach, but you never know :-).

> > Am I seeing things correctly?
> Yes I think so otherwise.
> > There are other 2 important aspects (cpython c modules and java
> > integration), but that will come a bit later.
> CPython c modules are tough -- for now we translate them into Java.
> Someday we may look to calling C extensions through JNA as Charles
> Nutter of JRuby has considered doing in the Ruby realm, but that is
> perhaps pie in the sky talk.

Well, if there are important modules that people may miss right away
then you/we should definitely consider
porting them to Java. Is there such a list available? The only
reference I could find is around 2 years old already.

We should formalize a list, but we're working slightly differently now. JRuby inspired us to tackle whole applications; I credit Eric Dobbs for bringing this approach to my attention. So that's why we're interested in Django or ipython or TurboGears 2; not only are these useful apps to run on Jython, they help prioritize this translation process and expose gaps that simple porting would not have.

From a methodology perspective, such apps function like a customer in an agile setting.

Of course we can do better and more, but I think recent work has demonstrated that we are making really good progress.

- Jim

Jim Baker

Jim Baker