jython-dev

[Jython-dev] moduleDictInit is not called

[Samuele P. <pe...@in...>]

Hi.

First I should apologize if my patch is a little bit confusing.

I tried the last commited version of the patch that should allow to load =
compiled python modules
and get them as working modules in the interp (based on invocation of =
moduleDictInit).

Given my patch, the point where  moduleInitDict is called is somehow =
wrong.
The point is that I have left createFromClass(String,InputStream) in the =
code
but this is no longer called and so createFromClass(String,Class) for =
classes
coming from sys.path.
This make indeed sense: java classes are treated more symmetrically =
indipendently from where they come
as we wanted but so moduleInitDict is not called when supposed. Clearly =
here we want to able
to break the symmetry when this make sense and we are loading a compiled =
module.
The problem is that given the patch there is no clear, coherent point =
where to put=20
moduleDictInit invocation.
My (not deeply tested) suggestion to get the desired functionality is =
not to use=20
moduleDictInit but simply to repeat the behaviour used when executing =
compiled
code outside of the interp:
the interp when looking for x should try not only x.py and x$py.class =
but
also x$_PyInner.class and then in this last case create an instance , =
getMain and execute
the obtained code in the dict of a new module.
Let me know...=20
I can experiment with this.

regards, Samuele

Re: [Jython-dev] moduleDictInit is not called

[<bc...@wo...>]

On Sat, 18 Nov 2000 06:20:22 +0100, you wrote:

>First I should apologize if my patch is a little bit confusing.

Not at all. I had not thoroughly checked the execution path of the new
code.

>I tried the last commited version of the patch that should allow to load =
>compiled python modules
>and get them as working modules in the interp (based on invocation of =
>moduleDictInit).
>
>Given my patch, the point where  moduleInitDict is called is somehow =
>wrong.
>The point is that I have left createFromClass(String,InputStream) in the =
>code
>but this is no longer called and so createFromClass(String,Class) for =
>classes
>coming from sys.path.
>This make indeed sense: java classes are treated more symmetrically =
>indipendently from where they come
>as we wanted but so moduleInitDict is not called when supposed. Clearly =
>here we want to able
>to break the symmetry when this make sense and we are loading a compiled =
>module.

My *intention* for moduleInitDict was as a general hook which any java
class could implement if it wanted some control over its __dict__ when
it is imported. It was not only for compiled modules.

>The problem is that given the patch there is no clear, coherent point =
>where to put=20
>moduleDictInit invocation.

Right. It is a problem for the lazy loaded classes.

>My (not deeply tested) suggestion to get the desired functionality is =
>not to use=20
>moduleDictInit but simply to repeat the behaviour used when executing =
>compiled
>code outside of the interp:
>the interp when looking for x should try not only x.py and x$py.class =
>but
>also x$_PyInner.class and then in this last case create an instance , =
>getMain and execute
>the obtained code in the dict of a new module.

That will allow initialization of the module dict when importing the
module and at the same time avoid the initialization when the proxy is
used from python. In addition type(x) would be ModuleType instead of
TypeType.

So all together, your suggestion sounds better.

Still, I feel a link is missing. Some way to go from x.class to
x$_PyInner.class, maybe as some public API in imp.java or Py.java.

regards,
finn

Re: [Jython-dev] moduleDictInit is not called

[Samuele P. <pe...@in...>]

Hi.

[Finn]
> My *intention* for moduleInitDict was as a general hook which any java
> class could implement if it wanted some control over its __dict__ when
> it is imported. It was not only for compiled modules.
For that we have classInitDict, which invocation seems to me in the right
place.
Clearly classDictInit is ok for manipulating the __dict__ but should not run
a module main.
Why the InitModule and his support code are still left around?
Should we clean that?

> >My (not deeply tested) suggestion to get the desired functionality is =
> >not to use=20
> >moduleDictInit but simply to repeat the behaviour used when executing =
> >compiled
> >code outside of the interp:
> >the interp when looking for x should try not only x.py and x$py.class =
> >but
> >also x$_PyInner.class and then in this last case create an instance , =
> >getMain and execute
> >the obtained code in the dict of a new module.
>
> That will allow initialization of the module dict when importing the
> module and at the same time avoid the initialization when the proxy is
> used from python. In addition type(x) would be ModuleType instead of
> TypeType.
>
> So all together, your suggestion sounds better.
I think we should put such code that deal with $_PyInner classes
in loadBuiltin and loadFromPath. ((The builtins coming from compilation
 exceptions and cPickle_exc ... should then be better recompiled)).
But then I imagine that a user would expect to be able to put
a jar containing compiled modules in classpath and be able
to import them. The simplest straightforward impl
will not offer that, limiting the search of $_PyInner classes
to sys.path as for .py and $py.class files.
I do not know if can offer the latter with a clean semantics,
but we can use for that openResourceAsStream.
I'm also asking myself if we should consider the dynamic value
of __path__ when looking for $_PyInner modules or not.
(( Maybe all that is related to the big design question, wheter we want
to load .py($py.class) also from classpath and have jars
on sys.path... ))
Notice that with this design choice if someone load a compiled
module through a java package it will get simply the java class
not the module, for me this makes sense but then we remain
with the problem of how to enable the usage of compiled py classes
as java classes from jython. You have always declared that this
is really difficult and I should admit that I have not studied it yet,
can you list what are the serious problem with that?

> Still, I feel a link is missing. Some way to go from x.class to
> x$_PyInner.class, maybe as some public API in imp.java or Py.java.
Sorry, I do not understand what  you mean with that.

regards, Samuele

Re: [Jython-dev] moduleDictInit is not called

[<bc...@wo...>]

[Samuele Pedroni]

>Hi.
>
>[Finn]
>> My *intention* for moduleInitDict was as a general hook which any java
>> class could implement if it wanted some control over its __dict__ when
>> it is imported. It was not only for compiled modules.
>For that we have classInitDict, which invocation seems to me in the right
>place.
>Clearly classDictInit is ok for manipulating the __dict__ but should not run
>a module main.
>Why the InitModule and his support code are still left around?
>Should we clean that?

I have not removed it in the small hope that 3rd part modules which uses
InitModule would continue to run with jython. I have not tested this, so
this kind of compatibility may have been broken by some other change.

>> So all together, your suggestion sounds better.

>I think we should put such code that deal with $_PyInner classes
>in loadBuiltin and loadFromPath. ((The builtins coming from compilation
> exceptions and cPickle_exc ... should then be better recompiled)).
>But then I imagine that a user would expect to be able to put
>a jar containing compiled modules in classpath and be able
>to import them. The simplest straightforward impl
>will not offer that, limiting the search of $_PyInner classes
>to sys.path as for .py and $py.class files.

Right.

>I do not know if can offer the latter with a clean semantics,
>but we can use for that openResourceAsStream.
>I'm also asking myself if we should consider the dynamic value
>of __path__ when looking for $_PyInner modules or not.
>(( Maybe all that is related to the big design question, wheter we want
>to load .py($py.class) also from classpath and have jars
>on sys.path... ))
>Notice that with this design choice if someone load a compiled
>module through a java package it will get simply the java class
>not the module, for me this makes sense 

I fear it will be FAQ. In some setups you will get the module, in others
you will get the proxy. They have the same name so it is not obvious
which you happened to import.

>but then we remain
>with the problem of how to enable the usage of compiled py classes
>as java classes from jython. 

I would not mind too much if the generated proxy classes wasn't
available from jython at all. As long as the module and all the classes
inside the module are available.

>You have always declared that this
>is really difficult and I should admit that I have not studied it yet,
>can you list what are the serious problem with that?

One problem that I have looked at is the way proxies stop their search
for attributes when they meet their first java superclass. As a result
subclasses of proxies can not find methods in the superclass and
overriding does not work:

------ x.py ------
import java
class x(java.lang.Object):
   def equals(self, other):
       print "equals", self, other
       return 0
------ END ------

Compile the x.py module to java class x.java.


>>> import x
>>> print type(x)
org.python.core.PyJavaClass
>>> a = x()
>>> print a
x@682b72
>>> print a.equals(a), a.equals(x)
1 0
>>>



>> Still, I feel a link is missing. Some way to go from x.class to
>> x$_PyInner.class, maybe as some public API in imp.java or Py.java.

>Sorry, I do not understand what  you mean with that.

I was thinking about the case where x.class and x$_PyInner.class exist
in a java package. A call to imp.load("x") will get the x.class, but
what if the programmer wanted the x.py module. Basicly the same problem
you described above.

Hmmm....

I think we should load the x.class when found in a java package. We can
decide if we should change this when we get more feedback from users.
So loadBuiltin and loadFromPath should look for $_PyInner as the last
option.

regards,
finn

BTW, the exceptions.java and cPickle_exceptions modules are just a
temporary hack. And so should the loadBuiltin be IMO. I have included an
old email where I try to describe the reasoning for the hack:


Subject: [Jython-dev] builtin exceptions
From: bc...@wo... (Finn Bock)
Date: Sat, 30 Sep 2000 19:32:09 GMT

Hi,

There is a problem with the exceptions module. At least there is if we
want to follow CPython's route where:

- string based std. exception are dropped.
- the exceptions module is builtin.

I think we want both. And when there no longer are string based std
exception as a fall back, we must be very certain that the exceptions
module can be loaded. So we also want to make it a builtin module.

I'd tried different experiments in the erratas. First I coded the
exceptions module as java class:

package org.python.modules;
public class exceptions  {
    public static class Exception extends PyObject {
        public static String __doc__ =
            "Base class for all standard Python exceptions.";

        public PyTuple args = Py.EmptyTuple;

        public Exception() {
        }

        public Exception(PyObject[] args, String[] kws) {
           this.args = new PyTuple(args);
        }

        public PyString __str__() {
            switch (args.__len__()) {
            case 0:
                return Py.newString("");
            case 1:
                return args.__getitem__(0).__str__();
            default:
                return args.__str__();
            }
        }

...
}

Initially this worked like a charm. From python this type can be
extended as usual and from java it can be extended as

    public static class PickleError extends exceptions.Exception {
        public PickleError() { }

        public PickleError(PyObject[] args, String[] kws) {
            super(args, kws);
        }

        public PyString __str__() {
            if (args.__len__() > 0 && args.__getitem__(0).__len__() > 0)
                return args.__getitem__(0).__str__();
            else
                return new PyString("(what)");
        }
    }



It does have some problems:

1) type(Exception) return TypeType, not ClassType

2) A java class does not allow changes to its __name__, nor can it
   control how it will be represented by str() and repr(). (Instances 
   of a java class can control it's representation, but the class
   itself can't).

3) A python class will not be able to inherit from two such exceptions.


Based on the experience, I decided to give up on exceptions.java and
instead built the exceptions module as a java class which create the
python classes by hand. Just like the exceptions.c does.

Unfortunately JPython have no API designed for this. The only way you
can create a python class is the jpythonc (and the dynamic code
compiler) creates python classes. That API (PyTableCode) is best suited
for mechanical code generation.

So in the latest errata I used jpythonc on exceptions.py, made it create
the file org/python/modules/exceptions.java and included this class in
the jar file.

Yes, it's Bad (tm).

We have to design a human usable API with which a programmer can create
python classes and populate it with functions and attributes. I just
fear that it isn't achievable before the first alpha. 

Our short term options are, as I see them:

1) Keep exceptions.py *and* use string based std exception as a
   fallback. I'll hate that because it makes it difficult to 
   create modules that must work with both string based and class 
   based exceptions.

2) Keep exceptions.py *without* any fallback if exceptions.py fail
   to load.

3) Include the generated exceptions.java (and the similarly generated
   cPickle_exceptions.java). 

I favor number 3.

Re: [Jython-dev] importing compiled code

[Samuele P. <pe...@in...>]

Hi.

first some bugs status:
1) I have put in place the fix for [ Bug #122610 ] SecEx in MS applerviewer.
    http://sourceforge.net/bugs/?func=detailbug&bug_id=122610&group_id=12867
2) Bug [ Bug #122793 ] Calling reload on java class loaded from sys.path
cause NPE
    http://sourceforge.net/bugs/?func=detailbug&bug_id=122793&group_id=12867
    is clearly closed because actually reload for a jclass is nop,
   or open if we want to keep it as reminder that we should implement
reloading for jclasses <wink>.
3) [ Bug #122864 ] importing java packages
    http://sourceforge.net/bugs/?func=detailbug&bug_id=122864&group_id=12867
    has been closed by my patch that treats jars and dirs symmetrically.
4) [ Bug #122876 ] Import error with mixed classpath is solved by my patch
too.
    http://sourceforge.net/bugs/?func=detailbug&bug_id=122876&group_id=12867

2nd:
[Finn]
> >Why the InitModule and his support code are still left around?
> >Should we clean that?
>
> I have not removed it in the small hope that 3rd part modules which uses
> InitModule would continue to run with jython. I have not tested this, so
> this kind of compatibility may have been broken by some other change.
I think, it still works, e.g. PyDictionary still used it, I have just
changed that
but we can keep InitModule for 3rd part compatibility.

Now about importing compiled code:

I'm not convinced about *my* proposal:

[Finn]
> ...
> >Notice that with this design choice if someone load a compiled
> >module through a java package it will get simply the java class
> >not the module, for me this makes sense
>
> I fear it will be FAQ. In some setups you will get the module, in others
> you will get the proxy. They have the same name so it is not obvious
> which you happened to import.
> ...
>
> I would not mind too much if the generated proxy classes wasn't
> available from jython at all. As long as the module and all the classes
> inside the module are available.
>
I'm not conviced that what we can quickly(?) implement is what people
expect.
It seems to me that jythonc is more there to build applets or make jython
classes
resemble java classes, such that people can forget about their jythonic
nature.
If someone subclasses a swing panel or builds a bean with jythonc I believe
it expects a class
not a module when he imports that back in the interpreter.
I know this is just philosophy...
but there is also a technical side:
* as noticed the java class and the python module have the same name,
  this is really a problem, e.g w.r.t. to sys.modules
  I always felt that top-level jclasses should not end up in sys.modules but
that
  is what jython does. My viewpoint is that java classes are not modules...
* actually one can get to import the compiled class, but constructed
instances
  do not work, I have done few experiments and read the code a bit, so maybe
I'm missing something,
  but this point could be solved making the PyJavaClass.__call__ return a
python
  class in this case, here it does not make sense to put a proxy around a
proxy. We should take care of
  this also by subclassing. (see transcript at the end).
* but the most serious issue is the following and this is an issue
concerning both choices (class/pkg import):
   we should be careful  choosing with which classloader we load things, in
mixed java/jython code case
   it is quite is easy and natural use the pre-built proxy from the java
side (and this at java compilation time), I can imagine  that if   we are
not careful we risk to load things twice and get a new series of interop
problems ...
* what should reload do?

Are we in hurry to offer such a feature. Personally I need more time to
study the issue.
Could you [Finn] list what is left to do (up to bugs), what should be in for
alpha1 ... and the final release.
For example the cache issue as a step toward reloading - I think - is
important.

About that there are still some questions around that I should answer you.
I will post the answers together with a list of ideas (many are very
low-prio).

Comments are welcome.

regards, Samuele

----- an experiment
This just indicates that the other choice (class import) is not hopeless,
but this does not mean easy.

Setup:
<Z.py>
import java

class Z(java.lang.Object):
 def value(self):
  "@sig public int value()"
  return 1
</Z.py>
This gets compiled with jythonc to Z.class and Z$_PyInner.class.

<SZ.java>
public class SZ {
 public static int svalue(Z z) {
  return z.value();
 }
}
</SZ.java>
This gets compiled with javac against Z.class (from jythonc) to SZ.class.

The bad news:
Jython 2.0 pre-alpha on java1.3.0 (JIT: null)
Type "copyright", "credits" or "license" for more information.
>>> import Z
>>> z=Z()
>>> z.value()
Traceback (innermost last):
  File "<console>", line 1, in ?
AttributeError: abstract method "value" not implemented

The "good" news:
Jython 2.0 pre-alpha on java1.3.0 (JIT: null)
Type "copyright", "credits" or "license" for more information.
>>> import sys
>>> from java.lang import Class
>>> import Z,SZ
>>> (Z.getClassLoader(),SZ.getClassLoader()) # things loaded from sys.path
(org.python.core.BytecodeLoader@ae1cf, org.python.core.BytecodeLoader@ae1cf)
>>> sys.modules['Z']
<jclass Z at 8321686>
>>> del sys.modules['Z'] # otherwise we get an exception
>>> z=Class.newInstance(Z)
>>> z
<Z.Z instance at 3180698>
>>> z.value()
1
>>> sys.modules['Z']
<module Z at 4920025>
>>> z1=Class.newInstance(Z)
>>> sys.modules['Z']
<module Z at 4920025>
>>> (SZ.svalue(z),SZ.svalue(z1))
(1, 1)
>>> class Y(Z):
...  pass
...
>>> y=Y()
>>> y.value()
Traceback (innermost last):
  File "<console>", line 1, in ?
AttributeError: abstract method "value" not implemented
>>> # but
>>> class Y(z.__class__):
...  pass
...
>>> y=Y()
>>> y.value()
1
>>> # and most important
>>> SZ.svalue(y)
1
>>> # seems to work.

Re: [Jython-dev] importing compiled code

[<bc...@wo...>]

On Mon, 20 Nov 2000 02:11:58 +0100, you wrote:

>Hi.
>
>first some bugs status:
>1) I have put in place the fix for [ Bug #122610 ] SecEx in MS applerviewer.
>    http://sourceforge.net/bugs/?func=detailbug&bug_id=122610&group_id=12867
>2) Bug [ Bug #122793 ] Calling reload on java class loaded from sys.path
>cause NPE
>    http://sourceforge.net/bugs/?func=detailbug&bug_id=122793&group_id=12867
>    is clearly closed because actually reload for a jclass is nop,
>   or open if we want to keep it as reminder that we should implement
>reloading for jclasses <wink>.

Let us close as many bug reports as we can. If we want to have
remainders we can always open a new one with a more accurate
description.

>Now about importing compiled code:
>
>I'm not convinced about *my* proposal:
>
>[Finn]
>> ...
>> >Notice that with this design choice if someone load a compiled
>> >module through a java package it will get simply the java class
>> >not the module, for me this makes sense
>>
>> I fear it will be FAQ. In some setups you will get the module, in others
>> you will get the proxy. They have the same name so it is not obvious
>> which you happened to import.
>> ...
>>
>> I would not mind too much if the generated proxy classes wasn't
>> available from jython at all. As long as the module and all the classes
>> inside the module are available.
>>
>I'm not conviced that what we can quickly(?) implement is what people
>expect.
>It seems to me that jythonc is more there to build applets or make jython
>classes
>resemble java classes, such that people can forget about their jythonic
>nature.
>If someone subclasses a swing panel or builds a bean with jythonc I believe
>it expects a class
>not a module when he imports that back in the interpreter.

That is all true. I just don't think we can fullfill that expectation in
the more complicated situations:

=========== test260s1.py =========== 
import java
class test260s1(java.util.Vector):
   def foo(self):
       pass

class P(java.awt.Panel):
   pass

class foo:
   pass
=========== END =========== 

If the main code isn't run at some time during import, all the python
code (foo above) is lost. It is my believe (at the moment anyway) that
is is easier to explain that the test260s1 class is in fact called
test260s1.test260s1 then it is to explain what exactly that test260s1
thing is in the first place.

>I know this is just philosophy...
>but there is also a technical side:
>* as noticed the java class and the python module have the same name,
>  this is really a problem, e.g w.r.t. to sys.modules
>  I always felt that top-level jclasses should not end up in sys.modules but
>that
>  is what jython does. My viewpoint is that java classes are not modules...
>* actually one can get to import the compiled class, but constructed
>instances
>  do not work, I have done few experiments and read the code a bit, so maybe
>I'm missing something,
>  but this point could be solved making the PyJavaClass.__call__ return a
>python
>  class in this case, here it does not make sense to put a proxy around a
>proxy. 

Yes, that could be a likely solution.

> We should take care of
>  this also by subclassing. (see transcript at the end).
>* but the most serious issue is the following and this is an issue
>concerning both choices (class/pkg import):
>   we should be careful  choosing with which classloader we load things, in
>mixed java/jython code case
>   it is quite is easy and natural use the pre-built proxy from the java
>side (and this at java compilation time), I can imagine  that if   we are
>not careful we risk to load things twice and get a new series of interop
>problems ...
>* what should reload do?

He. I dunno, but if anyone wants to use reloading on a compiled python
class they shouldn't have compiled it in the first place.

>Are we in hurry to offer such a feature. 

No. I added it because I though it would be a small and easy change.
I'll remove my ModuleDictInit changes. 

>Personally I need more time to
>study the issue.
>Could you [Finn] list what is left to do (up to bugs), what should be in for
>alpha1 ... and the final release.
>For example the cache issue as a step toward reloading - I think - is
>important.
>
>About that there are still some questions around that I should answer you.
>I will post the answers together with a list of ideas (many are very
>low-prio).
>
>Comments are welcome.
>
>regards, Samuele
>
>----- an experiment
>This just indicates that the other choice (class import) is not hopeless,
>but this does not mean easy.
>
>Setup:
><Z.py>
>import java
>
>class Z(java.lang.Object):
> def value(self):
>  "@sig public int value()"
>  return 1
></Z.py>
>This gets compiled with jythonc to Z.class and Z$_PyInner.class.
>
><SZ.java>
>public class SZ {
> public static int svalue(Z z) {
>  return z.value();
> }
>}
></SZ.java>
>This gets compiled with javac against Z.class (from jythonc) to SZ.class.
>
>The bad news:
>Jython 2.0 pre-alpha on java1.3.0 (JIT: null)
>Type "copyright", "credits" or "license" for more information.
>>>> import Z
>>>> z=Z()
>>>> z.value()
>Traceback (innermost last):
>  File "<console>", line 1, in ?
>AttributeError: abstract method "value" not implemented
>
>The "good" news:
>Jython 2.0 pre-alpha on java1.3.0 (JIT: null)
>Type "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> from java.lang import Class
>>>> import Z,SZ
>>>> (Z.getClassLoader(),SZ.getClassLoader()) # things loaded from sys.path
>(org.python.core.BytecodeLoader@ae1cf, org.python.core.BytecodeLoader@ae1cf)
>>>> sys.modules['Z']
><jclass Z at 8321686>
>>>> del sys.modules['Z'] # otherwise we get an exception
>>>> z=Class.newInstance(Z)
>>>> z
><Z.Z instance at 3180698>
>>>> z.value()
>1
>>>> sys.modules['Z']
><module Z at 4920025>
>>>> z1=Class.newInstance(Z)
>>>> sys.modules['Z']
><module Z at 4920025>
>>>> (SZ.svalue(z),SZ.svalue(z1))
>(1, 1)
>>>> class Y(Z):
>...  pass
>...
>>>> y=Y()
>>>> y.value()
>Traceback (innermost last):
>  File "<console>", line 1, in ?
>AttributeError: abstract method "value" not implemented
>>>> # but
>>>> class Y(z.__class__):
>...  pass
>...
>>>> y=Y()
>>>> y.value()
>1
>>>> # and most important
>>>> SZ.svalue(y)
>1
>>>> # seems to work.

So maybe the ability to use the proxies directly from jython isn't so
far away.

Rather than trying to decide if "z" should be a module or a class, we
should give the user the option of choosing. Maybe as additional options
to jythonc where the user can specify which of classes what should
become main proxies (main proxies are real java classes, ordinary
proxies are innerclasses). By default the module name will be importable
as a module unless it is overridden as a main proxy. So

  jythonc z.py

would create a z.class proxy and no module, and

  jython --mainproxy="" z.py

would create a z.class module and a z$z.class proxy.

There is probably better and more natural ways of specifing this, but I
hope you get the general idea.

regards,
finn