Re: [Jython-dev] poor man freezing and loading from other sources: proposals and ideas

[<bc...@wo...>]

[Samuele]

>Hi.
>
>[Finn]
>> >Please, no pure "it would be nice if" requests.
>> 
>> But it would be so nice, if we had all this <wink>.
>Maybe not all in time for 2.1, but in principle <wink>.
>Let's get a bit concrete.
>
>> Opening the sys.classLoader API would be a good thing, but it also opens
>> the question of "scanning" the sys.classLoader for java packages. We
>> could perhaps make it a requirement that sys.classLoader.getPackages()
>> should return something usefull. Unfortunately I think most custom
>> classloaders ignore the package support completely.
>In any case getPackages() is protected, and my interpretation of the docs
>is that even a good-willing classloader can limit itself to define packages 
>lazily only when classes are loaded from them.

Yeah.

>On the other hand even if the only solution is to define packages
>through sys.add_* apis, I think that opening the sys.classLoader API
>and making imports even more regular is something necessary for many embedding 
>related situations, in any case it is something not for the casual user, the 
>others can pay the necessary attention and lack of comfort...

Yes, at least in principle. It's just that I can already hear the
complaints that jython can't load java class: "I know the classloader
works because I can load classes from java". It is painfull that the
complaint is actually valid.

Maybe a different worded ImportError should be printed when
sys.classLoader is set.

>> > iv) <poor man freezing>
>...
>> > idea: have a tool that take a jar or a set of directories
>> > and collect which java packages-classes/python packages-modules are defined
>> > and produce a report in a textual format (for maximal flexibility).
>> >
>> > This report can be put in the jar and be read (impl/expl) by jython runtime
>> > as a _resource_  in    
>> > order to initiliaze its internal tables for special and java loading (see 
>*)).
>> 
>> A tool sound fine to me. Having such an additional build step is truly a
>> minor thing.
>> 
>> > open issues: $py.class files still require that proxies and adapters
>> > are built at runtime, so classloader creation permission is required,
>> > this should not be an issue for server side deployment or trusted
>> > applets, for normal applets jythonc solve this. Is this a serious 
>> > problem?
>> 
>> I'm sure it depends on the deployment environment, whether or not it is
>> a problem <wink>.
>> 
>> An idea could be a runtime option that caused proxies and adapters to be
>> saved in user specified directory. It would also cause the $py.class
>> file to be rewritten with a compiled-in proxy reference. That could
>> perhaps maintain the illusion that we don't have to pre-compile the
>> application.
>There is something in that direction (at least for adapters) for debugging 
>purpose. I think what you propose is doable. Maybe we can even
>avoid to put explicit references (and have 2 $py.class formats),
>at least for adapters the actual code works both for the interpreter
>case and jythonc compiled code without using explicit refs. We
>can try to extend and adapt this.

I think listeners are less likely to be edited, and when they are
changed, the loading/linking of the adapter will most likely fail.

Proxies are much more fragile since changes to both the java class and
the python subclass can/should force a new proxy class to be generated.
But to avoid two $py.class formats, we could always compile in the proxy
classname and only use the name when running as a poor-man frozen
application.

>Clearly we cannot support classes redefined multiple times or defined
>only along one execution path but nobody is asking for that or I'm wrong?
><wink>

At least that is easy to explain and to suggest workarounds if any users
have such expectations.

>And yes the tool solution is still the best I can conceive.
>
>> >v) eventually jreload.makeLoadSet can be extended to accept
>> > a classloader (this will not allow reloading but makes sense for
>> > the loading from other srcs issue) 
>> 
>> My senses tells me that this is a minor corner. The real issue for other
>> load sources and sys.classLoader when jython is embedded.
>> 
>Ok, I will not do that. For the moment I have received 1 positive
>feedback about jreload and the tiny bug report. Nothing personal
>but without feeback (+,-,?) I will leave it untouched.
>
>> >i) allow to have jars specified in sys.path from which both
>> >   java classes or python modules can be imported.
>> > open issues:
>...
>> 
>> >  - should the jars added to sys.path be kept open for performance
>> >    reasons as long as they are in sys.path?
>> 
>> Dunno. I suppose it depend on the actual performance of opening an
>> archive, but I would expect the boost to be significant.
>> 
>> >  - when should the "slow" scan for java packages be scheduled, when
>> >    the jar is added to sys.path (more deterministic) or at the next 
>> >    import, should the results be cached (IMHO no), should there be
>> >    a tool to prescan a jar and store the information? in the
>> >    normal jython cache? or together or inside the jar?
>> 
>> Have you seen the "-i" option to the JDK1.3 jar command? I hadn't until
>> now. It seems very nifty. I may change my mind later, but for a first
>> attempt I feel that it is OK to require that some kind of indexing is
>> run on the .jar files on sys.path. Printing a message for non-indexed
>> jars:
>> 
>>    "xxx.jar is ignored. Please run the jarindex command on xxx.jar"
>> 
>Unfortunaly jar -i does not store enough information to support
>dir and import *. In any case we would have to build a tool for
>non java SE2 1.3 env (?).

Yes. I called this tool of ours for "jarindex". It would be a shell/.bat
script written by the installer.

>To be honest this one  is the more complicated of the proposed changes,
>here some concrete ideas:
>
>0. have a tool to "index" a jar in a way that supports dir (maybe the same
>  as above under different options)

I believe that a .jar should only be scanned once. That means that if
the scan result isn't cached we should reject un-indexed jars. Why don't
you want to cache the scan result?

All in all this is my favorite.

>a. ignore performance issues, open and close jars on every import, scan
> or read the index also lazily on import

There shouldn't be noticeable performance difference between putting a
jar on sys.path or on classpath. I think we have to keep it open.

>b. like a) but keep jars open, but I don't see any good place to lazily
>   close them that work at least in many situations if not for all.
>   Just keeping them open sounds bad.

That is what the JVM does for the .zip files found on CLASSPATH. As for
changes to sys.path se below.

>c. the overkill solution [maybe not that much but implies that we must live
>   forever with the following CPython incompatibility (which is quite marginal):
>      L=UserList.UserList()
>      sys.path=L
>   works in CPython but not in Jython, for c) we will have to even enforce this]
>   add to builtin lists the possibility to have an observer (it costs
>   the comparision between a ref and a null on every update method)
>   that is notified for changes, so we can open, scan and close things
>   on sys.path changes. Why every list? to make rare things work:
>      L=['.','foodir']
>      sys.path=L
>      L.append('bazdir')
>   When a list is assigned to sys.path its observer field is assigned,
>   the field of the old one is reset...

Or keep a copy of the sys.path internally. Whenever we want to access
Py.getSystemState().path from java we compare the actual list with the
internal copy and deal with the differences. That would close the
archive at the next import.

Or wrap the list 'L' above with an observable wrapper that intercepts
__XXXitem__ calls. That would close the archive immeditatly but cause
the value returned by 'sys.path' to be different from the value
assigned. My favorite solution.

>d. the "ugly" solution
>   sys.jar_open('foobaz.jar') # it is also scanned or index read
>   sys.path.append('foobaz.jar')
>   ...
>   sys.path.remove('foobaz.jar')
>   sys.jar_close('foobaz.jar')

No!

>   one can put a jar in sys.path without opening it => the jar
>   will be opened and closed on each import, fair enough ;)
>
>All of them have pros and cons, b) needs an answer to the open question,
>c) is somehow "user-friendly", maybe even too much and maybe should be
>discussed with python-dev people because if CPython takes the decision
>of extending sys.path usage, this too implicit approach can be difficult
>to port to the new situation. 

I suppose that putting the zipfile instance itself into sys.path would
solve the close problem. Until CPython decides to go the full length, I
would prefer a less neat solution where we keep only strings in
sys.path.

regards,
finn