Re: [Jython-dev] work in progress: cache

[Samuele P. <pe...@in...>]

Hi.


> >Some questions:
> >
> >I would like to distinguish the support for dir() for java packages from
> >the support for from <pkg> import *. I dislike the fact that dir
> >force lazy loading, because __dir__ is used by from import * code.
> >Lazy loading should be limited to the from import * case.
> >The fact is that I want to minimize the "potential" robustness issues
> >of the - in any case - necessary cache patch.
> >Any remark or hint on this is welcome?
> 
> In principle, I don't see a problem if the "dir()" builtin on a java
> package initialized the java classes, but your argumentation for doing
> so is weak. The cache code must be robust in all cases. If it isn't, its
> a bug that must be fixed.
> 
> Implementationwise, both "from xx import *" and "dir()" uses the
> __dir__() hook, so do you propose make them diffrent?
> 

The patch that I'm working on do not contain the cache fix.

It works as follows wrt java package:

PyJavaPackage through findattr give access to all the classes in the package

but the internal __dict__ contains only the actually loaded classes.

My idea is that dir() returns the list of all names of classes but this not
implies that they are actually in the internal __dict__,
Maybe my idea is bad or will probably break a lot of code, but
I imagine there are few people doing:
 import jpkg
 
 for clname in dir(jpkg):
  print jpkg.__dict__[clname]
  
Any suggestion is welcome.

About robustness of the cache patch:

1) without the patch we cannot have reloading
2) I once written something about the technical aspects of what I imagine
 a patch could be, I repeat it here, kindly asking for feedback and opinions.
 The point is that 100% robustness seems to me not possible,
 because we have to make assumptions about what lies behind the classloaders
 that we just receive in some sense without having control over them.
 But I agree that we should make it works without errors also in
 presence of very nasty classloader context, if one avoids  from jpkg import *.

> This will introduce the cache fixing issue:
> 
> i) we cannot just use a Class->PyJavaClass cache, because this will
> not support lazy loading (from jpkg import *).
> 
> ii) there are 2 options
>   1) just a (classloader,classname)->PyJavaClass cache
>   2) a Class->PyJavaClass and a cache like 1) that should be kept 
synchronized.
>      The (classloader,classname) cache will only be filled with classes
>      that can incur in lazy loading.
>      
>   (1) is simpler to implent, but less efficient than (2)
>   (2) is more complicated and error-prone
>   
> iii) the main problem is that we should be able to anticipate which classes
> can incur in lazy loading and their "classloaders":
> 
>   jclass=loader.loadClass("...)
> 
> in general we will not have jclass.getClassLoader()==loader (*)
> 
> In java2 the classloaders are organized in parent-child hierarchies,
+ but the programmer can build also other relations,
> with java11 classloaders world is flat, but this does not mean that
> (*) always holds.
> 
> If we can have full control on things (f.ex. the load sets) we can
> assume (*) to be true but that's not the case in general:
> a way to proceed is the following:
> 
> when we anticipate the "classloader" setting up things for lazy loading
> we anticipate loader.
> 
> But when we receive a loaded jclass we should be able to map its classloader
> back to the guesses we make in the lazy loading setup case. And we have
> also just the classloader to decide if a class can incur in lazy loading
> because classes can just arrive from the pure java side of the world.
> 
> In order to do this we should make some assumptions (no solution in this
> context can be strictly robust, with (2) we can add an option
> that disable lazy loading and so the guess machinary when one knows
> that things can go wrong) and should work for the common and
> usual cases.
> 
> What we need is a way to guess and a mapping from actual classloaders
> to the guesses:
> here is a proposal and discussion starting point:
> 
> I assume that sys.classLoader is not a hook (but in this case we should
> better protect it) and is set only once, if this is not the case things
> become more complicated (maybe impossible).
> 
>  normal lazy loading (the one that we have now and that will remain): 
>   guess = None // null
>  lazy loading from the context of a load sets (in the future):
>   guess = classloader-of-the-load-set
>   // the framework should force (*) to be true
>   
>   
>  mapping: 
>    we need the helper: 
>     
>     fromhier(jclass,classloader):
>      java11:
>       return jclass.classLoader == classloader or jclass.classLoader == None
>      java2:
>       try:
>        cl=jclass.classLoader
>       except <security>:
>        return 1
+       if cl==None:
+        return 1
>       try:
>        while 1:
>         if cl == classloader:
>          return 1
+         if classloader==None:
+             return 0
>         classloader = classloader.parent
>       except <security>:
>        pass
>       return 0
> 
>    sys.classLoader is sys.classLoader
>    rtloader is the loader of Python runtime (Py.class.getClassLoader())
>       
>    the_mapping(jclass):
>     if fromhier(jclass,rtloader) or fromhier(jclass,sys.classLoader):
>      return None
>     if jclass.classLoader is a load-set-classloader:
>      return it
>     return 'cannot incur in lazy loading'

regards, Samuele