Re: [Jython-dev] poor man freezing and loading from other sources: proposals and ideas

[Samuele P. <pe...@in...>]

Hi.

[Finn]
> >Please, no pure "it would be nice if" requests.
> 
> But it would be so nice, if we had all this <wink>.
Maybe not all in time for 2.1, but in principle <wink>.
Let's get a bit concrete.

> Opening the sys.classLoader API would be a good thing, but it also opens
> the question of "scanning" the sys.classLoader for java packages. We
> could perhaps make it a requirement that sys.classLoader.getPackages()
> should return something usefull. Unfortunately I think most custom
> classloaders ignore the package support completely.
In any case getPackages() is protected, and my interpretation of the docs
is that even a good-willing classloader can limit itself to define packages 
lazily only when classes are loaded from them.
On the other hand even if the only solution is to define packages
through sys.add_* apis, I think that opening the sys.classLoader API
and making imports even more regular is something necessary for many embedding 
related situations, in any case it is something not for the casual user, the 
others can pay the necessary attention and lack of comfort...

> > iv) <poor man freezing>
...
> > idea: have a tool that take a jar or a set of directories
> > and collect which java packages-classes/python packages-modules are defined
> > and produce a report in a textual format (for maximal flexibility).
> >
> > This report can be put in the jar and be read (impl/expl) by jython runtime
> > as a _resource_  in    
> > order to initiliaze its internal tables for special and java loading (see 
*)).
> 
> A tool sound fine to me. Having such an additional build step is truly a
> minor thing.
> 
> > open issues: $py.class files still require that proxies and adapters
> > are built at runtime, so classloader creation permission is required,
> > this should not be an issue for server side deployment or trusted
> > applets, for normal applets jythonc solve this. Is this a serious 
> > problem?
> 
> I'm sure it depends on the deployment environment, whether or not it is
> a problem <wink>.
> 
> An idea could be a runtime option that caused proxies and adapters to be
> saved in user specified directory. It would also cause the $py.class
> file to be rewritten with a compiled-in proxy reference. That could
> perhaps maintain the illusion that we don't have to pre-compile the
> application.
There is something in that direction (at least for adapters) for debugging 
purpose. I think what you propose is doable. Maybe we can even
avoid to put explicit references (and have 2 $py.class formats),
at least for adapters the actual code works both for the interpreter
case and jythonc compiled code without using explicit refs. We
can try to extend and adapt this.
Clearly we cannot support classes redefined multiple times or defined
only along one execution path but nobody is asking for that or I'm wrong?
<wink>
And yes the tool solution is still the best I can conceive.

> >v) eventually jreload.makeLoadSet can be extended to accept
> > a classloader (this will not allow reloading but makes sense for
> > the loading from other srcs issue) 
> 
> My senses tells me that this is a minor corner. The real issue for other
> load sources and sys.classLoader when jython is embedded.
> 
Ok, I will not do that. For the moment I have received 1 positive
feedback about jreload and the tiny bug report. Nothing personal
but without feeback (+,-,?) I will leave it untouched.

> >i) allow to have jars specified in sys.path from which both
> >   java classes or python modules can be imported.
> > open issues:
...
> 
> >  - should the jars added to sys.path be kept open for performance
> >    reasons as long as they are in sys.path?
> 
> Dunno. I suppose it depend on the actual performance of opening an
> archive, but I would expect the boost to be significant.
> 
> >  - when should the "slow" scan for java packages be scheduled, when
> >    the jar is added to sys.path (more deterministic) or at the next 
> >    import, should the results be cached (IMHO no), should there be
> >    a tool to prescan a jar and store the information? in the
> >    normal jython cache? or together or inside the jar?
> 
> Have you seen the "-i" option to the JDK1.3 jar command? I hadn't until
> now. It seems very nifty. I may change my mind later, but for a first
> attempt I feel that it is OK to require that some kind of indexing is
> run on the .jar files on sys.path. Printing a message for non-indexed
> jars:
> 
>    "xxx.jar is ignored. Please run the jarindex command on xxx.jar"
> 
Unfortunaly jar -i does not store enough information to support
dir and import *. In any case we would have to build a tool for
non java SE2 1.3 env (?).

To be honest this one  is the more complicated of the proposed changes,
here some concrete ideas:

0. have a tool to "index" a jar in a way that supports dir (maybe the same
  as above under different options)
a. ignore performance issues, open and close jars on every import, scan
 or read the index also lazily on import
b. like a) but keep jars open, but I don't see any good place to lazily
   close them that work at least in many situations if not for all.
   Just keeping them open sounds bad.
c. the overkill solution [maybe not that much but implies that we must live
   forever with the following CPython incompatibility (which is quite marginal):
      L=UserList.UserList()
      sys.path=L
   works in CPython but not in Jython, for c) we will have to even enforce this]
   add to builtin lists the possibility to have an observer (it costs
   the comparision between a ref and a null on every update method)
   that is notified for changes, so we can open, scan and close things
   on sys.path changes. Why every list? to make rare things work:
      L=['.','foodir']
      sys.path=L
      L.append('bazdir')
   When a list is assigned to sys.path its observer field is assigned,
   the field of the old one is reset...
d. the "ugly" solution
   sys.jar_open('foobaz.jar') # it is also scanned or index read
   sys.path.append('foobaz.jar')
   ...
   sys.path.remove('foobaz.jar')
   sys.jar_close('foobaz.jar')

   one can put a jar in sys.path without opening it => the jar
   will be opened and closed on each import, fair enough ;)

All of them have pros and cons, b) needs an answer to the open question,
c) is somehow "user-friendly", maybe even too much and maybe should be
discussed with python-dev people because if CPython takes the decision
of extending sys.path usage, this too implicit approach can be difficult
to port to the new situation. Suggestions?

regards.