Re: [Jython-dev] poor man freezing and loading from other sources: proposals and ideas

[<bc...@wo...>]

[Samuele]

>Hi.
>
>Jython 2.0 more clean semantic wrt import and java loading
>was an improvement over JPython. 

That's an understatement!

>Now often more features have been requested in the directions of what has
>been called:
>
>- "poor man freezing" (the ability to take interpreter $py.class files
>  and package them somehow for deployment)
>- (java) loading from other sources (the possibility to have
>   import working from jars or classloaders specified at runtime) .
>
>Here are some ideas and proposals toward something concrete in those
>directions for 2.1.
>
>Note: jython 2.0 'jreload' module in principle can be used to enable
> imports from jars at runtime.
>
>i) allow to have jars specified in sys.path from which both
>   java classes or python modules can be imported.
> open issues:
>  - will that produce comp. problem with pre-existent code
>    that expects only dirs in sys.path?  

I doubt it. Existing uses of sys.path shouldn't blow up if a directory
is renamed or removed.

>  - should the jars added to sys.path be kept open for performance
>    reasons as long as they are in sys.path?

Dunno. I suppose it depend on the actual performance of opening an
archive, but I would expect the boost to be significant.

>  - when should the "slow" scan for java packages be scheduled, when
>    the jar is added to sys.path (more deterministic) or at the next 
>    import, should the results be cached (IMHO no), should there be
>    a tool to prescan a jar and store the information? in the
>    normal jython cache? or together or inside the jar?

Have you seen the "-i" option to the JDK1.3 jar command? I hadn't until
now. It seems very nifty. I may change my mind later, but for a first
attempt I feel that it is OK to require that some kind of indexing is
run on the .jar files on sys.path. Printing a message for non-indexed
jars:

   "xxx.jar is ignored. Please run the jarindex command on xxx.jar"

>  Some of this behaviour require sys.path changes to have side effects,
>  this could be difficult to obtain in a clean way.
>  
>ii) with i) jython -jar foo.jar will put foo.jar in sys.path ;)

Yes.

>iii) make jython behaviour even more regular:
>   (some definitions:
>     rt-classloader: is the classloader of jython runtime,
>     sys.classLoader is sys.classLoader ;) )
>   actually jython behaviour is as follows:
>   sys.classLoader not set:
>     java loading: rt-classloader > sys.path
>     python loading: sys.path
>   sys.classLoader set:
>     java loading: sys.classLoader
>     python loading: sys.path
>   special case: jythonc compiled code (=> sys.classLoader set)
>     java loading rt-classloader
>     python loading: special from rt-classloader > sys.path
>     but in this case as required sys.path==[]
>     
>   ideas:
>     applicative code should be able to set sys.classLoader
>     obtaining a meaningful behaviour (typically this should not
>     be done or at most once but this should not be enforced),
>     there should be a version of PythonInterpreter.initialize
>     that takes a classLoader and set it in sys.classLoader
>     
>   new behaviour: always:
>    java loading: rt-classloader > sys.classLoader (if set) > sys.path

Opening the sys.classLoader API would be a good thing, but it also opens
the question of "scanning" the sys.classLoader for java packages. We
could perhaps make it a requirement that sys.classLoader.getPackages()
should return something usefull. Unfortunately I think most custom
classloaders ignore the package support completely.

>    python loading: special* from rt-classloader & sys.classLoader > sys.path
>
>   *) by now jythonc compiled code comes with an hardcoded list that specifies
>     which python modules/packages should be loaded from java classloaders,
>     this should be mantained and extended, see iv)
>     
> iv) <poor man freezing>
> how to deal with java -jar foo.jar where foo.jar contains jython runtime,
> java classes using PythonInterpreter API and a set of packaged $py.class
> or .py files ; or the similar applet case, etc ?
>
> First it should be noted that in general we cannot assume that we can scan
> the jar at runtime.
> 
> idea: have a tool that take a jar or a set of directories
> and collect which java packages-classes/python packages-modules are defined
> and produce a report in a textual format (for maximal flexibility).
>
> This report can be put in the jar and be read (impl/expl) by jython runtime
> as a _resource_  in    
> order to initiliaze its internal tables for special and java loading (see *)).

A tool sound fine to me. Having such an additional build step is truly a
minor thing.

> open issues: $py.class files still require that proxies and adapters
> are built at runtime, so classloader creation permission is required,
> this should not be an issue for server side deployment or trusted
> applets, for normal applets jythonc solve this. Is this a serious 
> problem?

I'm sure it depends on the deployment environment, whether or not it is
a problem <wink>.

An idea could be a runtime option that caused proxies and adapters to be
saved in user specified directory. It would also cause the $py.class
file to be rewritten with a compiled-in proxy reference. That could
perhaps maintain the illusion that we don't have to pre-compile the
application.

>Note: the open issue is somehow related  (could be partially solved by) to the 
>possibility of loading jythonc compiled code in the interpreter (this actually 
>fails), some of the proposals here could help to construct a framework for that
>but for the moment I'm not addressing such a situation.
>
>
>v) eventually jreload.makeLoadSet can be extended to accept
> a classloader (this will not allow reloading but makes sense for
> the loading from other srcs issue) 

My senses tells me that this is a minor corner. The real issue for other
load sources and sys.classLoader when jython is embedded.

> and load-sets could have add_package- like features.

Yes, but it is a poor workaround when compared to automatic java package
scanning.

>I would appreciate some feedback about what is/is not necessary,
>design ideas and input about the open issues.
>
>Please, no pure "it would be nice if" requests.

But it would be so nice, if we had all this <wink>.

regards,
finn