jython-dev

Re: [Jython-dev] importing compiled code

[Samuele P. <pe...@in...>]

Hi.

[Finn]
> So maybe the ability to use the proxies directly from jython isn't so
> far away.
> 
> Rather than trying to decide if "z" should be a module or a class, we
> should give the user the option of choosing. Maybe as additional options
> to jythonc where the user can specify which of classes what should
> become main proxies (main proxies are real java classes, ordinary
> proxies are innerclasses). By default the module name will be importable
> as a module unless it is overridden as a main proxy. So
> 
>   jythonc z.py
> 
> would create a z.class proxy and no module, and
> 
>   jython --mainproxy="" z.py
> 
> would create a z.class module and a z$z.class proxy.
> There is probably better and more natural ways of specifing this, but I
> hope you get the general idea.

Yes, good point.
From my viewpoint the main problem we are dealing with are the name 
clashes, so if we want the new feature is better to revert the old practice:
user compiles a set of modules and marks through jythonc cmd-line options or
"@sig"-like docs for which classes he wants to get pre-comp proxies,
in principle he should avoid name clashes between proxies and modules, and
such clashes are errors.
But to be backward compatible if there is "just one" of such clashes,
a proxie for the involved class (e.g MyBean) is produced with the right name
and behind the scene the enclosing module is renamed (MyBean -> MyBean_home),
the assumption here is actually that people with the current usage of jythonc
ignore the module at all, clearly a warning should be issued.
2nd: the fact that actually things (module and proxie) are packaged together
as inner class and class is nice, but if we can create more than a proxie 
from a single py this does not work anymore. From the technical side
a more loose coupling is quite better (also in single proxie case) w.r.t. to the 
classloaders issues.
All this involves changes to runtime and jythonc, for the moment I would
prefer keep it low-prio.

Please, what are the plans for alpha1 and up to final 2.0?
From my side I should finish clarify the cache issue. And with such
a patch it should be possible to produce an experimental version
of reloading support (in form of a python module using __import__ hook).

regards, Samuele.

Re: [Jython-dev] importing compiled code

[<bc...@wo...>]

[Samuele on proxyclasses and module]

>All this involves changes to runtime and jythonc, for the moment I would
>prefer keep it low-prio.

I agree.

>Please, what are the plans for alpha1 and up to final 2.0?

Featurewise I think we have what we need for alpha1. I have some further
tests I want to perform on my own applications and I hope others also
try the version in CVS on some real world application and applets.

Apart from the code we need

- A license.
- There are still some bugs in the installer. (not including the latest
  bugs I introduced in pa3+4+5. I now know what caused them)
- Update some of the text files: NEWS and README.

Before beta I would like to:

- Fix some of the bugs when running CPython's test suite.
- Support the ucnhash and unicodedata modules.

>From my side I should finish clarify the cache issue. 

Right. I don't think it can get into alpha1 but the hooks to manage the
the caches should certainly go in before we enter beta.

>And with such
>a patch it should be possible to produce an experimental version
>of reloading support (in form of a python module using __import__ hook).

Good. I would like to get broader feedback on the reloading issues, so
maybe some version of java reloading can go into 2.0-final marked as
experimental. I would also like Guido's view on how pythonic he consider
our java reload support to be. 

regards,
finn

Re: [Jython-dev] catchall

[Samuele P. <pe...@in...>]

Hi.

*  First.
[Finn]
> [Samuele on proxyclasses and module]
>
> >All this involves changes to runtime and jythonc, for the moment I would
> >prefer keep it low-prio.
>
> I agree.
Ok, let  us postpone that, but please, I would like to hear your opinion on
my last proposals
about clashes resulting in errors, and backward compatibility.

[Finn]
> - Update some of the text files: NEWS and README.
Both are important given your errata, the python 2.0 compatibility, and the
many broken things that now should work,
and the new load design with its rules.

An important note:
my patch ignores extendedClassLoader option, sys.path java loading is always
on, because having classpath taking over
should already avoids all problems. But it is still in the registry with a
meaningless comment and handled by Option.java...

* Random notes on the cache issue and reloading (things I can work on):
(1)
[me]
>From my side I should finish clarify the cache issue.
Here are the pointers to last mails on this:
[me] http://www.geocrawler.com/lists/3/SourceForge/7018/50/4602714/
[Finn] http://www.geocrawler.com/lists/3/SourceForge/7018/50/4606179/

Sorry if my explanations were confused,
when I wrote
> in general we will not have jclass.getClassLoader()==loader (*)
I was making a general statement about java class loading where 'loader' is
some java.lang.ClassLoader,
and not an example of our reload support.
We can surely support both examples imagined by Finn.
The robustness issue is not directly related to reloading, and in any case
we can set up some minimal
requirements to classloaders passed to reload support. But if we want
reloading we should fix the cache;
and the cache fix I can imagine has a robustness issue related to
*uniqueness* of PyJavaClasses when
we have from ... import * lazy loading.

I repeat this is related to uniqueness of PyJavaClasses not directly to
reloading,
and this is an example when things can go wrong.
The setup is the following
sys.classLoader is set to some foreign classloader (for example our code
is some jythonc compiled code running under an application server (e.g.
servlet container,...)).
The code does from jpkg import *, jpkg contains e.g. a class Bean.
At this point we put a lazy PyJavaClass pjc0 in our cache with key
(null,"jpkg.Bean"),
null meaning that we guess the class will come from runtime/sys.classLoader
(or its parents).
Now our code receives an instance inst0 of jpkg.Bean from java side of the
world,
and should find a corresponding PyJavaClass for it. Actually inst0.getName()
== 'jpkg.Bean'.
For cl:=inst0.getClass()
 sys.classLoader.loadClass("jpkg.Bean") == cl
but cl.getClassLoader() is not sys.classLoader nor one of its parents.
This is really a bad way to put up classloaders given the java2 rules but
can happen (?).
Now we are unable to link this instance with the pjc0 because it does not
seem that
its classloader come from sys.classLoader hierarchy => so we create a second
PyJavaClass
pjc1 != pjc0 and we lose uniqueness.
That's is in some sense the only possible bad scenario, not really the
common case and we have
problem just because of lazy import *.
(2) I think it would be great to have both unloading support (meaning
discarding things from internal
jython classes related caches) through explicit calls but also (optionally)
under java2 lazily/implicitly through
the use of weak-refs. I imagine a pluggable architecture for the caches.
Should not be too difficult.
(3) For reloading also MakeProxies.makeClass should be changed:
   - it leaks memory, blocking (together with the caches) proxies unloading,
because
     the direct proxies for classpath/sys.path loaded classes are defined
directly in syspathJavaLoader.
   - it does not support proxies with parents coming from classloaders
different from syspathJavaLoader
     and its ancestors,
   this has already been written: we should use loaders for proxies with a
local (direct parents) class pool to solve that,
   this should work because there should be no intrapackage relationships
issues (hope so ;) )
  Maybe it is better to distinguish between a class implementing sys.path
loading and a class for dynamic
  loading of discardable code. Actually BytecodeLoader does both things.

[Finn]
> >And with such
> >a patch it should be possible to produce an experimental version
> >of reloading support (in form of a python module using __import__ hook).
>
> Good. I would like to get broader feedback on the reloading issues, so
> maybe some version of java reloading can go into 2.0-final marked as
> experimental. I would also like Guido's view on how pythonic he consider
> our java reload support to be.
Hope that our best is enough ;), in any case if we should keep our reload
support
separate from runtime (because it is not enough pythonic) we will encouter
the need
of some ihooks/imputil-like support and suffer the destiny of import-sig,
which has mostly ignored jython, otherwise people will not be able to hook
importing for other purposes.

* A framework for extending importing would also be great for jython, for
example for poor-man freezing,
 putting support for everything directly in the runtime being not a good
choice.

* jythonc side
+ package relative import has not been fixed yet.
+ proposed --depend option.
+ for the future:
  - importing of compiled code into interp, and compiling code referring to
separate comp results.
  - (maybe) jythonc able to compile against a claspath/sys.path different
from the classpath/sys.path under which is running

* (mostly a potential users wish) it is really difficult to effectively use
java security arch from jython and
  for jython code,  some kind of rexec mixing java/python policies is maybe
a necessary framework. (very complicated)
* performance issues?(?)

regards, Samuele.

Re: [Jython-dev] catchall

[<bc...@wo...>]

[Samuele Pedroni]

>* Random notes on the cache issue and reloading (things I can work on):
>(1)
>[me]
>>From my side I should finish clarify the cache issue.
>Here are the pointers to last mails on this:
>[me] http://www.geocrawler.com/lists/3/SourceForge/7018/50/4602714/
>[Finn] http://www.geocrawler.com/lists/3/SourceForge/7018/50/4606179/
>
>Sorry if my explanations were confused,
>when I wrote
>> in general we will not have jclass.getClassLoader()==loader (*)
>I was making a general statement about java class loading where 'loader' is
>some java.lang.ClassLoader,
>and not an example of our reload support.
>We can surely support both examples imagined by Finn.
>The robustness issue is not directly related to reloading, and in any case
>we can set up some minimal
>requirements to classloaders passed to reload support. But if we want
>reloading we should fix the cache;
>and the cache fix I can imagine has a robustness issue related to
>*uniqueness* of PyJavaClasses when
>we have from ... import * lazy loading.
>
>I repeat this is related to uniqueness of PyJavaClasses not directly to
>reloading,
>and this is an example when things can go wrong.
>The setup is the following
>sys.classLoader is set to some foreign classloader (for example our code
>is some jythonc compiled code running under an application server (e.g.
>servlet container,...)).
>The code does from jpkg import *, jpkg contains e.g. a class Bean.
>At this point we put a lazy PyJavaClass pjc0 in our cache with key
>(null,"jpkg.Bean"),
>null meaning that we guess the class will come from runtime/sys.classLoader
>(or its parents).
>Now our code receives an instance inst0 of jpkg.Bean from java side of the
>world,
>and should find a corresponding PyJavaClass for it. Actually inst0.getName()
>== 'jpkg.Bean'.
>For cl:=inst0.getClass()
> sys.classLoader.loadClass("jpkg.Bean") == cl
>but cl.getClassLoader() is not sys.classLoader nor one of its parents.
>This is really a bad way to put up classloaders given the java2 rules but
>can happen (?).
>Now we are unable to link this instance with the pjc0 because it does not
>seem that
>its classloader come from sys.classLoader hierarchy => so we create a second
>PyJavaClass
>pjc1 != pjc0 and we lose uniqueness.
>That's is in some sense the only possible bad scenario, not really the
>common case and we have
>problem just because of lazy import *.

Let us for a moment say that we disable lazy loading. Each class in the
class list will then be resolved (through Py.findClassEx I think) into a
Class instance and the PyJavaClass will be complete including the link
from Class to PyJavaClass in PyJavaClass.classes.

When we in this theoretical senary receive the inst0 instance from the
java side its class either match an entry in PyJavaClass.classes and if
it does not match, we create a new entry with inst.getClass() as key.

So, lets add lazy loading. We now have an PyJavaClass.classes dict that
looks something like this:

  {
     org.python.core.PyJavaClass.class : PyJavaClass(PyJavaClass@1),
     "jpkg.Bean" : PyJavaClass(null),
  }

meaning that the PyJavaClass is fully initialized with its .class as key
and that jpkg.Bean is lazily loaded and uninitialized with the string
"jpkg.Bean" as key.

When we get a java instance inst0 from the java side we first check for
a lazy entry. We find jpkg.Bean which is then resolved using the same
classloaders setup as if the class was not lazy loaded. When the
PyjavaClass is initialized the lazy entry is removed and the dict looks
like this:

  {
     org.python.core.PyJavaClass.class : PyJavaClass(PyJavaClass@1),
     jpkg.Bean.class@2 : PyJavaClass(jpkg.Bean.class@2),
  }

Now the inst0.getClass() is searched in the dict. If it happens to be
the same as jpkg.Bean.class@2, the newly initialized PyJavaClass is
returned. If it isn't, a new entry is added.

  {
     org.python.core.PyJavaClass.class : PyJavaClass(PyJavaClass@1),
     jpkg.Bean.class@2 : PyJavaClass(jpkg.Bean.class@2),
     jpkg.Bean.class@3 : PyJavaClass(jpkg.Bean.class@3),
  }

This last dictionary would match the situation where we had disabled
lazy loading. The two jpkg.Bean classes would not be interoperable, but
lazily loaded class would work just like the ordinarily initialized
classes.

I still don't have a deep and intuitive understanding of the issues, so
there is likely something I have forgotten.

>(2) I think it would be great to have both unloading support (meaning
>discarding things from internal
>jython classes related caches) through explicit calls but also (optionally)
>under java2 lazily/implicitly through
>the use of weak-refs. I imagine a pluggable architecture for the caches.
>Should not be too difficult.

Cool.

>(3) For reloading also MakeProxies.makeClass should be changed:
>   - it leaks memory, blocking (together with the caches) proxies unloading,
>because
>     the direct proxies for classpath/sys.path loaded classes are defined
>directly in syspathJavaLoader.
>   - it does not support proxies with parents coming from classloaders
>different from syspathJavaLoader
>     and its ancestors,
>   this has already been written: we should use loaders for proxies with a
>local (direct parents) class pool to solve that,
>   this should work because there should be no intrapackage relationships
>issues (hope so ;) )

Right.

>  Maybe it is better to distinguish between a class implementing sys.path
>loading and a class for dynamic
>  loading of discardable code. Actually BytecodeLoader does both things.

Yes, that would be goodness. Much confusion in the past have its roots
in this double duty.

>[Finn]
>> >And with such
>> >a patch it should be possible to produce an experimental version
>> >of reloading support (in form of a python module using __import__ hook).
>>
>> Good. I would like to get broader feedback on the reloading issues, so
>> maybe some version of java reloading can go into 2.0-final marked as
>> experimental. I would also like Guido's view on how pythonic he consider
>> our java reload support to be.
>Hope that our best is enough ;), in any case if we should keep our reload
>support
>separate from runtime (because it is not enough pythonic) we will encouter
>the need
>of some ihooks/imputil-like support and suffer the destiny of import-sig,
>which has mostly ignored jython, otherwise people will not be able to hook
>importing for other purposes.
>
>* A framework for extending importing would also be great for jython, for
>example for poor-man freezing,
> putting support for everything directly in the runtime being not a good
>choice.
>
>* jythonc side
>+ package relative import has not been fixed yet.
>+ proposed --depend option.
>+ for the future:
>  - importing of compiled code into interp, and compiling code referring to
>separate comp results.
>  - (maybe) jythonc able to compile against a claspath/sys.path different
>from the classpath/sys.path under which is running
>
>* (mostly a potential users wish) it is really difficult to effectively use
>java security arch from jython and
>  for jython code,  some kind of rexec mixing java/python policies is maybe
>a necessary framework. (very complicated)
>* performance issues?(?)

I agree with all of this. I'm unsure how we should maintain this list.
The CPython people have their PEP-0042 for this. 


regards,
finn

Re: [Jython-dev] importing compiled code

[<bc...@wo...>]

On Mon, 20 Nov 2000 15:24:42 +0100 (MET), you wrote:

>From my viewpoint the main problem we are dealing with are the name 
>clashes, so if we want the new feature is better to revert the old practice:

Ok, I'm not in any way married to the existing default behaviour of
jythonc.

>user compiles a set of modules and marks through jythonc cmd-line options or
>"@sig"-like docs for which classes he wants to get pre-comp proxies,
>in principle he should avoid name clashes between proxies and modules, and
>such clashes are errors.
>But to be backward compatible if there is "just one" of such clashes,
>a proxie for the involved class (e.g MyBean) is produced with the right name
>and behind the scene the enclosing module is renamed (MyBean -> MyBean_home),

No. Changing the module name is wrong. Changing the name without giving
the users control over the new name is twice wrong.

>the assumption here is actually that people with the current usage of jythonc
>ignore the module at all, clearly a warning should be issued.
>2nd: the fact that actually things (module and proxie) are packaged together
>as inner class and class is nice, but if we can create more than a proxie 
>from a single py this does not work anymore. 

It doesn't? It works in the cases I have tried. Only the MyBean.MyBean
class will become the main proxy. The other classes are usable from java
as MyBean.MyOtherClass.

Here is another suggestion.

The default for jythonc is changed so that the MyBean.class is a module
and the proxy for MyBean.MyBean is a public static innerclass.
There will be two ways of changing this default:

1) A new option "--mainproxy" which will cause the MyBean$MyBean.class
   proxy to be renamed to MyBean.class and the module is lost. So to 
   get the current behaviour for existing applets, the --mainproxy
   option must be specified. If the MyBean.MyBean class isn't a java
   class, a warning is issued. No warning is issued for the loss of the
   module, since it is done on behalf of the user.

2) Marking the class with the text "@mainproxy" in the class's
   doc-string. If the MyBean.MyBean class is marked this way, the 
   module is overriden and lost. No warning is issued for the loss of
   the module. If other classes in MyBean is marked with "@mainproxy",
   jythonc will create separate MyOtherClass.java source file, one for
   each "@mainproxy" marked class.

I think it is Ok to lose the module when it happens based on a
deliberate user choice. (There haven't been a great demand for the
module).

Changing the default will very likely become a faq: "Why doesn't my
applet work anymore", but I think we can answers these questions as they
are asked.

>From the technical side
>a more loose coupling is quite better (also in single proxie case) w.r.t. to the 
>classloaders issues.

Right now there is a strict coupling between the module and the proxies:
the proxies accessed through the module will be loaded by the same
classloaded that loaded the module. That coupling should probably be
loosened in some way, I haven't looked into that.

regards,
finn