jython-dev

[Jython-dev] Natively compiled jython, take II.

[Alexandre Petit-B. <apb...@cy...>]

OK. With a bit of help from hacking our runtime and trying things out,
I found out that if I have in my local directory all the files found
in Lib *and* a symlink called `java' pointing to the `java' directory
containing the class files hierarchy resulting from the build of our
clean-room Java run time, I can run testall.py.

There's something wrong with the way things are searched (if you have
insight on where in the code this is done, it would be cool -- I just
started to read PySystemState.java.) so I'm guessing our runtime
doesn't setup the same environment than a regular interpreter. It also
transpires in our VM (gij), since it runs things if the `java' symlink
is made (no needs for local copies of the *.py in that case.)

On the good side of things, testall reports good results. It crashed
in serialization because of bug in our runtime error most likely. And
then there where some other tests that I had to disable (for example,
exit doesn't behave the same way) but otherwise it brought me fairly
quickly to getting the same crash that I got with the install JDK
based version (ImportError: No module named AnonInner.)

Soon I hope to be able to hack the hooks to run the native compiler on
jython generated classes.

./A

Re: [Jython-dev] Natively compiled jython, take II.

[Samuele P. <pe...@in...>]

Hi

1. I know nothing about gcj rt.
2. I know a little about jython rt ;)

a) Unless you pass an explicit python.home property (in the sense of
System.getProperty)
to jython rt and if there is no real jython.jar, jython is unable to
locate the Lib dir
and put it in sys.path and to find the global registry. A user-specific
.jython registry
can be used as a workaround.

b) Does gcj has the java.class.path property somehow set.
   Jython rt tries to scan the jars and dirs on classpath in order to
list the java packages
  that are candidate for loading. Given what you describe it seems that
gcj classpath is
empty or contain at most the cwd. When you put the java symlink in the
cwd you enable
jython to do its scan. At the moment there is no direct workaround wrt to
gcj setting,
which - it seems - does not  implement/need a proper classpath for
classes that can be/are linked.

regards.

Re: [Jython-dev] Natively compiled jython, take II.

[Alexandre Petit-B. <apb...@cy...>]

Samuele Pedroni writes:

> to jython rt and if there is no real jython.jar, jython is unable to
> locate the Lib dir

Good point. I built my jar file like org-jython-lag.jar (lag for
latest and greatest, we'll switch over to 2.1 when it's out.) I never
crossed my mind that jython might be looking for its own jar. Duh.

> b) Does gcj has the java.class.path property somehow set.

Yes, it's set to the content of CLASSPATH (and maybe some other
things.) So with that pointing to our libgcj.jar archive, I don't need
the `java' symlink.

> Jython rt tries to scan the jars and dirs on classpath in order
> to list the java packages that are candidate for loading. Given what
> you describe it seems that gcj classpath is empty or contain at most
> the cwd.

Yes it was. Thanks!

So, bottom line. The problem was simply that I didn't have CLASSPATH
set properly. I didn't cross my mind that a extremely dynamic
environment like jython needed pointers to the archives it was going
to explore. Duh.

So when I position CLASSPATH to point to our runtime jar and the
distributed jython jar, jython works regardless of the location it was
started from.

However, if I use the jython jar we generated with gcj (gcj can also
go to bytecodes,) I get errors, but now that's my problem :-)

Thanks a lot everyone for your help!

./A

Re: [Jython-dev] Natively compiled jython, take II.

[<bc...@wo...>]

[Alexandre Petit-Bianco]

>OK. With a bit of help from hacking our runtime and trying things out,
>I found out that if I have in my local directory all the files found
>in Lib *and* a symlink called `java' pointing to the `java' directory
>containing the class files hierarchy resulting from the build of our
>clean-room Java run time, I can run testall.py.
>
>There's something wrong with the way things are searched (if you have
>insight on where in the code this is done, it would be cool -- I just
>started to read PySystemState.java.) so I'm guessing our runtime
>doesn't setup the same environment than a regular interpreter. It also
>transpires in our VM (gij), since it runs things if the `java' symlink
>is made (no needs for local copies of the *.py in that case.)

It sound like a problem in Jython's java package loading. Java package
loading is the feature that allow jython to do a 
   import java.awt
and know that java.awt is a java package. In a normal setup jython
detects that java.awt is a java package by scanning all the .jar (and
.zip) files it can find in the classpath. Directories in the .jar files
with .class files in it is indexesd as a java package. This index is
cached in {python.home}/cachedir.

This is all implemented in the *PackageManager.java files.

In addition to loading of java packages from .jar files, jython also
accept directories in the filesystem found on the CLASSPATH as java
packages. This is why your symlink helps.

I'm not at all sure how this scanning of .jar files can/should be done
in the gcj environment. Does the rt.jar (or classes.zip) file still
exists as a file? If it is replaced by something else (maybe a rt.so
library) is there any way to query its contents?

>On the good side of things, testall reports good results. It crashed
>in serialization because of bug in our runtime error most likely. And
>then there where some other tests that I had to disable (for example,
>exit doesn't behave the same way) but otherwise it brought me fairly
>quickly to getting the same crash that I got with the install JDK
>based version (ImportError: No module named AnonInner.)

The AnonInner.java source in Lib\test\javatests must be compiled
manually. The output .class files should just be placed the same
directory.

>Soon I hope to be able to hack the hooks to run the native compiler on
>jython generated classes.

How exactly would that work? Would a foo$py.class file be dynamicly
compiled into a foo$py.so library and loaded by the gij VM?

Do you have some web links that would allow us to read about the
enviroment that gij is running with?

regards,
finn

Re: [Jython-dev] Natively compiled jython, take II.

[Alexandre Petit-B. <apb...@cy...>]

Finn Bock writes:

> In addition to loading of java packages from .jar files, jython also
> accept directories in the filesystem found on the CLASSPATH as java
> packages. This is why your symlink helps.

And this is where setting CLASSPATH to point to our runtime jar and
jython's jar helps. Now I can run jython from anywhere.

> I'm not at all sure how this scanning of .jar files can/should be
> done in the gcj environment. Does the rt.jar (or classes.zip) file
> still exists as a file?

Yeah, it's a file called libgcj.jar.

> The AnonInner.java source in Lib\test\javatests must be compiled
> manually. The output .class files should just be placed the same
> directory.

Noted.

> How exactly would that work? Would a foo$py.class file be dynamicly
> compiled into a foo$py.so library and loaded by the gij VM?

Well, jython's executable would use class loading features of our VM
(that most importantly for us deals with loading things in and out,
without necessarily executing bytecode.) So basically, yes. The
mechanism is exposed here:

  http://gcc.gnu.org/cgi-bin/cvsweb.cgi/~checkout~/gcc/libjava/java/lang/natClassLoader.cc?rev=1.38&content-type=text/plain

(search for quux.) Note that this is a C++ file, as we seamlessly mix
native Java and C++ code -- that's how we write the native parts of
our runtime, but we support JNI if necessary. CNI is pretty cool, you
can read more about it here: http://gcc.gnu.org/java/papers/cni/t1.html

So you can write the following:

  Foo.java:
  class Foo {}

  jld.java:
  class jld {
    public static void main (String[] arg)
    {
        Class x;
        try {x = Class.forName ("Foo");} catch (Exception _) {return;}
        java.lang.reflect.Method m[] = x.getMethods();
        for (int i = 0; i < m.length; i++)
  	  System.out.println (m[i].toString());
    }
  }

Using gcj, you create an executable from jld.java:

 $ gcj jld.java --main=jld -o jld

If you execute it now, it will do nothing.

Compile Foo.java to a class file and you'll be able to extract
informations about the class Foo:
  
  $ gcj -C Foo.java
  $ ./jld
  public final void java.lang.Object.wait(long) throws ...
  public final void java.lang.Object.wait() throws ...

Now remove the classfile and generate the matching shared object:

  $ rm Foo.class
  $ gcj -fPIC -shared Foo.java  -o lib-Foo.so

(Or we could also do gcj -fPIC -shared Foo.class -o lib-Foo.so, which
 is what I hope to hook to jython and jythonc.)

Now re-run jld, it gives the same result because it's loading the
shared object in place of the class file. You can also compile
jld.java to a class file and let our interpreter do the work by
interpreting bytecodes (but still getting info about Foo loading a
shared object:)

  $ gcj -C jld.java
  $ gij jld
  ...

Note that libgcj's interpreter (gij) naturally naturally trades
bytecodes for shared object and applies it everywhere it can, as you
could make a shared library out of jld.java and let gij load it and
execute it natively instead of interpreting the bytecode:

  $ rm jld.class
  $ gcj -fPIC -shared jld.java -o lib-jld.so
  $ gij jld
  ...

Which still is not quite like having a jld stand alone executable, but
everything is still compiled natively.

> Do you have some web links that would allow us to read about the
> enviroment that gij is running with?

You can read the documentation section of our website:

  http://gcc.gnu.org/java/docs.html

We are fairly compliant with a mixture of 1.1, 1.2 and even 1.3. It's
a clean-room implementation of the Java runtime, we're working with the
classpath project. It's known to work on {x86,ia64,ppc,alpha}/linux,
*BSD is being re-fixed one more time, Solaris should work (it used to
be our primary platform but I'm not sure the interpreter works well
anymore.) Some people have been working on a Darwin port, and there
are also numerous lesser known CPU ports being hacked on.

./A