Re: [Jython-dev] catchall

[<bc...@wo...>]

[Samuele Pedroni]

>* Random notes on the cache issue and reloading (things I can work on):
>(1)
>[me]
>>From my side I should finish clarify the cache issue.
>Here are the pointers to last mails on this:
>[me] http://www.geocrawler.com/lists/3/SourceForge/7018/50/4602714/
>[Finn] http://www.geocrawler.com/lists/3/SourceForge/7018/50/4606179/
>
>Sorry if my explanations were confused,
>when I wrote
>> in general we will not have jclass.getClassLoader()==loader (*)
>I was making a general statement about java class loading where 'loader' is
>some java.lang.ClassLoader,
>and not an example of our reload support.
>We can surely support both examples imagined by Finn.
>The robustness issue is not directly related to reloading, and in any case
>we can set up some minimal
>requirements to classloaders passed to reload support. But if we want
>reloading we should fix the cache;
>and the cache fix I can imagine has a robustness issue related to
>*uniqueness* of PyJavaClasses when
>we have from ... import * lazy loading.
>
>I repeat this is related to uniqueness of PyJavaClasses not directly to
>reloading,
>and this is an example when things can go wrong.
>The setup is the following
>sys.classLoader is set to some foreign classloader (for example our code
>is some jythonc compiled code running under an application server (e.g.
>servlet container,...)).
>The code does from jpkg import *, jpkg contains e.g. a class Bean.
>At this point we put a lazy PyJavaClass pjc0 in our cache with key
>(null,"jpkg.Bean"),
>null meaning that we guess the class will come from runtime/sys.classLoader
>(or its parents).
>Now our code receives an instance inst0 of jpkg.Bean from java side of the
>world,
>and should find a corresponding PyJavaClass for it. Actually inst0.getName()
>== 'jpkg.Bean'.
>For cl:=inst0.getClass()
> sys.classLoader.loadClass("jpkg.Bean") == cl
>but cl.getClassLoader() is not sys.classLoader nor one of its parents.
>This is really a bad way to put up classloaders given the java2 rules but
>can happen (?).
>Now we are unable to link this instance with the pjc0 because it does not
>seem that
>its classloader come from sys.classLoader hierarchy => so we create a second
>PyJavaClass
>pjc1 != pjc0 and we lose uniqueness.
>That's is in some sense the only possible bad scenario, not really the
>common case and we have
>problem just because of lazy import *.

Let us for a moment say that we disable lazy loading. Each class in the
class list will then be resolved (through Py.findClassEx I think) into a
Class instance and the PyJavaClass will be complete including the link
from Class to PyJavaClass in PyJavaClass.classes.

When we in this theoretical senary receive the inst0 instance from the
java side its class either match an entry in PyJavaClass.classes and if
it does not match, we create a new entry with inst.getClass() as key.

So, lets add lazy loading. We now have an PyJavaClass.classes dict that
looks something like this:

  {
     org.python.core.PyJavaClass.class : PyJavaClass(PyJavaClass@1),
     "jpkg.Bean" : PyJavaClass(null),
  }

meaning that the PyJavaClass is fully initialized with its .class as key
and that jpkg.Bean is lazily loaded and uninitialized with the string
"jpkg.Bean" as key.

When we get a java instance inst0 from the java side we first check for
a lazy entry. We find jpkg.Bean which is then resolved using the same
classloaders setup as if the class was not lazy loaded. When the
PyjavaClass is initialized the lazy entry is removed and the dict looks
like this:

  {
     org.python.core.PyJavaClass.class : PyJavaClass(PyJavaClass@1),
     jpkg.Bean.class@2 : PyJavaClass(jpkg.Bean.class@2),
  }

Now the inst0.getClass() is searched in the dict. If it happens to be
the same as jpkg.Bean.class@2, the newly initialized PyJavaClass is
returned. If it isn't, a new entry is added.

  {
     org.python.core.PyJavaClass.class : PyJavaClass(PyJavaClass@1),
     jpkg.Bean.class@2 : PyJavaClass(jpkg.Bean.class@2),
     jpkg.Bean.class@3 : PyJavaClass(jpkg.Bean.class@3),
  }

This last dictionary would match the situation where we had disabled
lazy loading. The two jpkg.Bean classes would not be interoperable, but
lazily loaded class would work just like the ordinarily initialized
classes.

I still don't have a deep and intuitive understanding of the issues, so
there is likely something I have forgotten.

>(2) I think it would be great to have both unloading support (meaning
>discarding things from internal
>jython classes related caches) through explicit calls but also (optionally)
>under java2 lazily/implicitly through
>the use of weak-refs. I imagine a pluggable architecture for the caches.
>Should not be too difficult.

Cool.

>(3) For reloading also MakeProxies.makeClass should be changed:
>   - it leaks memory, blocking (together with the caches) proxies unloading,
>because
>     the direct proxies for classpath/sys.path loaded classes are defined
>directly in syspathJavaLoader.
>   - it does not support proxies with parents coming from classloaders
>different from syspathJavaLoader
>     and its ancestors,
>   this has already been written: we should use loaders for proxies with a
>local (direct parents) class pool to solve that,
>   this should work because there should be no intrapackage relationships
>issues (hope so ;) )

Right.

>  Maybe it is better to distinguish between a class implementing sys.path
>loading and a class for dynamic
>  loading of discardable code. Actually BytecodeLoader does both things.

Yes, that would be goodness. Much confusion in the past have its roots
in this double duty.

>[Finn]
>> >And with such
>> >a patch it should be possible to produce an experimental version
>> >of reloading support (in form of a python module using __import__ hook).
>>
>> Good. I would like to get broader feedback on the reloading issues, so
>> maybe some version of java reloading can go into 2.0-final marked as
>> experimental. I would also like Guido's view on how pythonic he consider
>> our java reload support to be.
>Hope that our best is enough ;), in any case if we should keep our reload
>support
>separate from runtime (because it is not enough pythonic) we will encouter
>the need
>of some ihooks/imputil-like support and suffer the destiny of import-sig,
>which has mostly ignored jython, otherwise people will not be able to hook
>importing for other purposes.
>
>* A framework for extending importing would also be great for jython, for
>example for poor-man freezing,
> putting support for everything directly in the runtime being not a good
>choice.
>
>* jythonc side
>+ package relative import has not been fixed yet.
>+ proposed --depend option.
>+ for the future:
>  - importing of compiled code into interp, and compiling code referring to
>separate comp results.
>  - (maybe) jythonc able to compile against a claspath/sys.path different
>from the classpath/sys.path under which is running
>
>* (mostly a potential users wish) it is really difficult to effectively use
>java security arch from jython and
>  for jython code,  some kind of rexec mixing java/python policies is maybe
>a necessary framework. (very complicated)
>* performance issues?(?)

I agree with all of this. I'm unsure how we should maintain this list.
The CPython people have their PEP-0042 for this. 


regards,
finn