Re: [Jython-dev] work in progress: cache

[<bc...@wo...>]

[Samuele Pedroni]

>> >Some questions:
>> >
>> >I would like to distinguish the support for dir() for java packages from
>> >the support for from <pkg> import *. I dislike the fact that dir
>> >force lazy loading, because __dir__ is used by from import * code.
>> >Lazy loading should be limited to the from import * case.
>> >The fact is that I want to minimize the "potential" robustness issues
>> >of the - in any case - necessary cache patch.
>> >Any remark or hint on this is welcome?

[me]

>> In principle, I don't see a problem if the "dir()" builtin on a java
>> package initialized the java classes, but your argumentation for doing
>> so is weak. The cache code must be robust in all cases. If it isn't, its
>> a bug that must be fixed.
>> 
>> Implementationwise, both "from xx import *" and "dir()" uses the
>> __dir__() hook, so do you propose make them diffrent?
>> 

[Samuele Pedroni]

>The patch that I'm working on do not contain the cache fix.
>
>It works as follows wrt java package:
>
>PyJavaPackage through findattr give access to all the classes in the package
>
>but the internal __dict__ contains only the actually loaded classes.
>
>My idea is that dir() returns the list of all names of classes but this not
>implies that they are actually in the internal __dict__,

This is OK. It is common that dir(o) returns more entries than exists in
the __dict__ (f.ex the contents of the __members__ and __methods__
attributes). 

Still, "dir(o)" results in a call to o.__dir__(), and "from x import *"
also results in a call to x.__dir__(). 

>Maybe my idea is bad or will probably break a lot of code, but
>I imagine there are few people doing:
> import jpkg
> 
> for clname in dir(jpkg):
>  print jpkg.__dict__[clname]

That is OK to break, since we at the same time add the workaround of
instead using:
  print getattr(jpkg, clname)

>Any suggestion is welcome.
>
>About robustness of the cache patch:
>
>1) without the patch we cannot have reloading
>2) I once written something about the technical aspects of what I imagine
> a patch could be, I repeat it here, kindly asking for feedback and opinions.

I didn't understand the problems and issues the first time around. I
still don't, but I will now at least try <wink>.

> The point is that 100% robustness seems to me not possible,

Then I think we should punt on it ..

> because we have to make assumptions about what lies behind the classloaders
> that we just receive in some sense without having control over them.

.. and make (or plan to make) the necessary hooks available so the
advanced reload programmer can add whatever assumptions that applies to
her application.

> But I agree that we should make it works without errors also in
> presence of very nasty classloader context, if one avoids  from jpkg import *.
>
>> This will introduce the cache fixing issue:
>> 
>> i) we cannot just use a Class->PyJavaClass cache, because this will
>> not support lazy loading (from jpkg import *).
>> 
>> ii) there are 2 options
>>   1) just a (classloader,classname)->PyJavaClass cache
>>   2) a Class->PyJavaClass and a cache like 1) that should be kept 
>synchronized.
>>      The (classloader,classname) cache will only be filled with classes
>>      that can incur in lazy loading.
>>      
>>   (1) is simpler to implent, but less efficient than (2)
>>   (2) is more complicated and error-prone
>>   
>> iii) the main problem is that we should be able to anticipate which classes
>> can incur in lazy loading and their "classloaders":
>> 
>>   jclass=loader.loadClass("...)
>> 
>> in general we will not have jclass.getClassLoader()==loader (*)

This false assumption seems to be at the core of the difficulties we
face. I'll have to make some example in order to understand when (*) is
false.


Example 1:

======= BEGIN X.java ======= 
public class X {
  public int value=6;
}
======= END ======= 

======= BEGIN Y.java ======= 
public class Y {
  public static void printX(X x) {
    System.out.println(x.value);
  }
}
======= END ======= 

>>>loader = jload.loader(".")
>>>print loader.__dict__
[]
>>>print loader.X
<jclass X at 11111>
>>>print loader.Y
<jclass Y at 22222>
>>>print loader.__dict__.keys()
['X', 'Y']
>>>loader.Y.printX(loader.X())
6
>>>loader.reload()
>>>print loader.__dict__.keys()
['X', 'Y']
>>>print loader.X
<jclass X at 33333>
>>>print loader.Y
<jclass Y at 44444>
>>>loader.Y.printX(loader.X())
6

The simplest example should work something like that and the assumption
(*) will always hold (correct me if I wrong!). The java classes that
gets resolved by the loader is always either loaded by the loader (and
can be reloaded) or is loaded by some system loader (and can't be
reloaded).

Example 1 covers uses such as rapid prototyping of java classes and test
drivers. It also includes IMO the majority of all request for java
reloading.



Example 2:

======= BEGIN Container.java ======= 
public interface Container {
  public void getName();
}
======= END ======= 

======= BEGIN Qcontainer.java ======= 
public class Qcontainer implemnts Container {
  public void getName() {
    return "Qcontainer";
  }
}
======= END ======= 

======= BEGIN lets/Qlet.java ======= 
package lets;
public class Qlet {
  public void doStart(Container contaner)
    System.out.println("My container is " + container.getName())
  }
}
======= END ======= 

>>>containerloader = jload.loader(".")
>>>container = containerloader.QContainer()
>>>containerloader.Container
<jclass Container at 11111>
#
# Here we tie together a list of loaders
>>>letsloader = jload.loader(".", containerloader)
>>>letsloader.lets.Qlet
<jclass lets.Qlet at 22222>
#
# Check if the resolved Container interface was loaded by the
# letsloader. It sholdn't be.
>>>print letsloader.__dict__.keys()
['lets.Qlet']
#
# The classes loaded by these two loaders are then interoperable.
>>>c = containerloader.QContainer()
>>>letsloader.lets.Qlet().doStart(c)
My container is QContainer
#
# This also holds after reloading the letsloader
>>>letsloader.reload()
>>>letsloader.lets.Qlet
<jclass lets.Qlet at 33333>
>>>letsloader.lets.Qlet().doStart(c)
My container is QContainer


If the containerloader is reloaded, I think all dependent loaders should
also be reloaded. Perhaps automaticly.

This example is modeled over the servlet idea. The QContainer is the
servlet server and the lets.Qlet is the servlet which can be reloaded
again and again.

>> In java2 the classloaders are organized in parent-child hierarchies,
>+ but the programmer can build also other relations,
>> with java11 classloaders world is flat, but this does not mean that
>> (*) always holds.
>> 
>> If we can have full control on things (f.ex. the load sets) we can
>> assume (*) to be true but that's not the case in general:

The situation where we do not have full control are?

- If the loaded java classes creates their own classloaders, and then
loads some classes which they wants to interoperable with classes loaded
by jython? Initially I have sympathy for this situation. Such a
situation does not have to work out of the box.
- Others? And please try to give example if you can.

>> a way to proceed is the following:
>> 
>> when we anticipate the "classloader" setting up things for lazy loading
>> we anticipate loader.
>> 
>> But when we recetive a loaded jclass we should be able to map its classloader
>> back to the guesses we make in the lazy loading setup case. And we have
>> also just the classloader to decide if a class can incur in lazy loading
>> because classes can just arrive from the pure java side of the world.
>> 
>> In order to do this we should make some assumptions (no solution in this
>> context can be strictly robust, with (2) we can add an option
>> that disable lazy loading and so the guess machinary when one knows
>> that things can go wrong) and should work for the common and
>> usual cases.

Of the top of my head (because I don't understand the situation where we
need to guess) I think we should punt on this. It is not like jython2.0
have to be the last word on java reloading. If we can handle the example
1 and maybe even example 2, I think we are multiple times better of then
all prevoiusly versions of JPython put together wrt. java reloading.

[example implementation snipped (again). I need to upgrade my background
and knowledge before I can evaluate it]

regards,
finn