jython-dev — Discussion of developing and enhancing Jython itself(!) (not how to use Jython)

[Jython-dev] Build tools...

[Dj <dj...@ru...>]

After grabbing jython, I threw a quick ant build.xml file together...
then I thought I'd better ask if there was actually a mandated build
procedure for
jython. I quite like the look of Ant as it keeps the portability up.

Dj

---- The build.xml as hacked up ------------

<project name="jython" default="compile" basedir=".">

<target name="init">
 <property name="sourceDir" value="org" />
 <property name="outputDir" value="classes" />
</target>

<target name="clean" depends="init">
 <deltree dir="${outputDir}" />
</target>

<target name="prepare" depends="clean">
 <mkdir dir="${outputDir}" />
</target>

<target name="compile" depends="prepare">
 <javac srcdir="${sourceDir}" destdir="${outputDir}" />
</target>

<target name="jar" depends="compile">
 <jar basedir="classes" jarfile="jython.jar" />
</target>

</project>

Re: [Jython-dev] listarchive and jython sources

[<bc...@wo...>]

[ulrich.schreiner]

>two simple questions:
>
>- i cannot browse the mailarchive. is there an archive? the link on SF 
>does not work. 

This is one of the problems with using an external service. It is no
work to use, but there are nothing we can do when it doesn't work. Let's
hope that geocrawler (or the DNS) recover soon.

>- my computer lives behind a firewall and i cannot connect to cvs via 
>pserver. is it possible to tgz'd the sources in a daily cron and make them 
>accessible via http?

In the CPython project, I think this is done by a volunteer, preferable
someone with access to a cron job and an always-on connection.

Anyone?

regards,
finn

[Jython-dev] listarchive and jython sources

[<ulr...@in...>]

hi,

two simple questions:

- i cannot browse the mailarchive. is there an archive? the link on SF 
does not work. 
- my computer lives behind a firewall and i cannot connect to cvs via 
pserver. is it possible to tgz'd the sources in a daily cron and make them 
accessible via http?

</usc>

Re: [Jython-dev] *loading: somehow last mail for this week - errata 2

[Samuele P. <pe...@in...>]

I apologize, I have written the last part of the last mail too quickly.

The (pseudo)code presented gives the idea, but it's a little bit buggy:

>  mapping:
>    we need the helper:
>
>     fromhier(jclass,classloader):
>      java11:
>       return jclass.classLoader == classloader or jclass.classLoader ==
None
>      java2:
>       try:
>        cl=jclass.classLoader
>       except <security>:
>        return 1
!special case cl is the java system classloader (null):
+       if cl==None:
+        return 1

>       if classloader == '<secured>':
>        return 0
!We don't need that.

>       try:
>        while 1:
>         if cl == classloader:
>          return 1

+         if classloader==None:
+             return 0

>         classloader = classloader.parent
>       except <security>:
>        pass
>       return 0

>    sys.classLoader is sys.classLoader
>    rtloader is the loader of Python runtime (Py.class.getClassLoader(),
>    '<secured>' if one cannot get it for security reasons in java2)
!Every class can ask for its own classloader no security problem at this
level :)

>
>    the_mapping(jclass):
>     if fromhier(jclass,rtloader) or fromhier(jclass,sys.classLoader):
>      return None
>     if jclass.classLoader is a load-set-classloader:
>      return it
>     return 'cannot incur in lazy loading'

regards, Samuele

Re: [Jython-dev] *loading: somehow last mail for this week - errata

[Samuele P. <pe...@in...>]

Sorry.

> ii) there are 2 options
>   1) just a (classloader,classname)->PyJavaClass cache
>   2) a Class->PyJavaClass and a cache like 1) that should be kept 
synchronized.
>      The (classloader,classname) cache will only be filled with caches
>      that can incur in lazy loading.
>      
>   (1) is simpler to implent, but less efficient than (2)
>   (2) is more complicated and error-prone

--> will only be filled with classes that

[Jython-dev] <test> sorry

[Samuele P. <pe...@in...>]

Re: [Jython-dev] *loading: somehow last mail for this week

[Samuele P. <pe...@in...>]

Hi.
This is very likely the last mail of this week.
From tomorrow up to monday I will not be online.
Maybe I will be able to read the mails through the mailing lists
pages at sourceforge.

This discussions has treated the future support for reloading,
solving the broken sys.path java loading, fixing compiler.*Maker bugs,
and defining a cleaner semantic for loading, in particular sys.path java
and python loading.

(a)
For the sys.path java loading there were(are) 2 options:
things should work as if sys.path were virtually appended to
CLASSPATH or we should use the dynamic __path__ values.

Finn and I agreed on the first option, and we have further developed it.
Barry?

Precedence should also be fixed.


(a''...')
[Finn]
> >But there is another issue: I have not yet analyzed it in depth.
> >
> >One can compile a set of py modules/py pkg under a user-defined
> >java pkg (package jythonc option) so for example py mod x.y
> >will be userpkg.x.y.
> >
> >If the code in interpreted mode load a java class x.JClass what should do
> >at runtime the same code when compiled and put in the pkg userpkg,
> >just still try to load x.JClass or userpkg.x.JClass, try both.
> 
> x.JClass IMO. The --package option to jpythonc should only be for the
> generated code, not for the entire world.

Rethinking about the problem I agree with you, just x.JClass.
There is no need to stress on compatibility for sys.path loading porting from 
interp. to compiled mode when one uses --package, this would be difficult
to achieve and I feel in any case confusing for somenone+bad.
This is also coherent with the principle that sys.path java loading
is just a feature for people in hurry and prototyping.

(b)
[Finn]
> I can make this change. Py.initProxy will then get a new "String[]
> modules" argument. I'll make it a patch,, i.e I will not commit this
> until we have settled the complete semantic change.

Great. (I'm sure you already know it) Also Py.runCode
and Py.initProperties - I think - are concerned by this.

(cache options: technical)
This will introduce the cache fixing issue:

i) we cannot just use a Class->PyJavaClass cache, because this will
not support lazy loading (from jpkg import *).

ii) there are 2 options
  1) just a (classloader,classname)->PyJavaClass cache
  2) a Class->PyJavaClass and a cache like 1) that should be kept synchronized.
     The (classloader,classname) cache will only be filled with caches
     that can incur in lazy loading.
     
  (1) is simpler to implent, but less efficient than (2)
  (2) is more complicated and error-prone
  
iii) the main problem is that we should be able to anticipate which classes
can incur in lazy loading and their "classloaders":

  jclass=loader.loadClass("...)

in general we will not have jclass.getClassLoader()==loader (*)

In java2 the classloaders are organized in parent-child hierarchies,
with java11 classloaders world is flat, but this does not mean that
(*) always holds.

If we can have full control on things (f.ex. the load sets) we can
assume (*) to be true but that's not the case in general:
a way to proceed is the following:

when we anticipate the "classloader" setting up things for lazy loading
we anticipate loader.

But when we receive a loaded jclass we should be able to map its classloader
back to the guesses we make in the lazy loading setup case. And we have
also just the classloader to decide if a class can incur in lazy loading
because classes can just arrive from the pure java side of the world.

In order to do this we should make some assumptions (no solution in this
context can be strictly robust, with (2) we can add an option
that disable lazy loading and so the guess machinary when one knows
that things can go wrong) and should work for the common and
usual cases.

What we need is a way to guess and a mapping from actual classloaders
to the guesses:
here is a proposal and discussion starting point:

I assume that sys.classLoader is not a hook (but in this case we should
better protect it) and is set only once, if this is not the case things
become more complicated (maybe impossible).

 normal lazy loading (the one that we have now and that will remain): 
  guess = None // null
 lazy loading from the context of a load sets (in the future):
  guess = classloader-of-the-load-set
  // the framework should force (*) to be true
  
  
 mapping: 
   we need the helper: 
    
    fromhier(jclass,classloader):
     java11:
      return jclass.classLoader == classloader or jclass.classLoader == None
     java2:
      try:
       cl=jclass.classLoader
      except <security>:
       return 1
      if classloader == '<secured>':
       return 0
      try:
       while 1:
        if cl == classloader:
         return 1
        classloader = classloader.parent
      except <security>:
       pass
      return 0

   sys.classLoader is sys.classLoader
   rtloader is the loader of Python runtime (Py.class.getClassLoader(),
   '<secured>' if one cannot get it for security reasons in java2)
      
   the_mapping(jclass):
    if fromhier(jclass,rtloader) or fromhier(jclass,sys.classLoader):
     return None
    if jclass.classLoader is a load-set-classloader:
     return it
    return 'cannot incur in lazy loading'
      

I have the feeling we have clarified a lot of things, but maybe still
missed something. Thanks.

I'm sorry to have produced just bad english and no line of code,
but I hope this will change in the near future.

regards, Samuele

[Jython-dev] NoClassDefFoundError exception shown.

[<bc...@wo...>]

Recently there have been several complaints about the silent handling of
primary the NoClassDefFoundError exception. To the user it look as if
the java class does not exists, even when it is obviously available.
Rather than silently catching the NoClassDefFoundError error, I would
prefer if the exception was returned as a java exception to the user.

The ClassNotFoundException exception is still used to signal that the
class is in fact not available.

ANy comments to this change of existing behaviour?


Index: Py.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/Py.java,v
retrieving revision 2.20
diff -c -r2.20 Py.java
*** Py.java	2000/10/13 20:02:59	2.20
--- Py.java	2000/10/19 18:36:45
***************
*** 556,561 ****
--- 556,581 ----
          }
      }
  
+ 
+     public static Class findClassEx(String name) {
+         try {
+             ClassLoader classLoader = Py.getSystemState().getClassLoader();
+             if (classLoader == null)
+                 return Class.forName(name);
+             else
+                 return classLoader.loadClass(name);
+         }
+         catch (ClassNotFoundException e) {
+             return null;
+         }
+         catch (IllegalArgumentException e) {
+             throw JavaError(e);
+         }
+         catch (LinkageError e) {
+             throw JavaError(e);
+         }
+     }
+ 
      private static void setArgv(String arg0, String[] args) {
          PyObject argv[] = new PyObject[args.length+1];
          argv[0] = new PyString(arg0);
Index: PyJavaClass.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/PyJavaClass.java,v
retrieving revision 2.21
diff -c -r2.21 PyJavaClass.java
*** PyJavaClass.java	2000/10/09 14:45:14	2.21
--- PyJavaClass.java	2000/10/19 18:36:49
***************
*** 74,80 ****
              return;
          initializing = true;
          if (proxyClass == null)
!             init(Py.findClass(__name__));
          init__bases__(proxyClass);
          init__dict__();
          initialized = true;
--- 74,80 ----
              return;
          initializing = true;
          if (proxyClass == null)
!             init(Py.findClassEx(__name__));
          init__bases__(proxyClass);
          init__dict__();
          initialized = true;
Index: PyJavaDirPackage.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/PyJavaDirPackage.java,v
retrieving revision 2.1
diff -c -r2.1 PyJavaDirPackage.java
*** PyJavaDirPackage.java	1999/05/17 19:59:39	2.1
--- PyJavaDirPackage.java	2000/10/19 18:36:49
***************
*** 63,69 ****
              return ret;
          }
  
!         Class c = Py.findClass(attrName);
          if (c != null) {
              ret = PyJavaClass.lookup(c);
              __dict__.__setitem__(name, ret);
--- 63,69 ----
              return ret;
          }
  
!         Class c = Py.findClassEx(attrName);
          if (c != null) {
              ret = PyJavaClass.lookup(c);
              __dict__.__setitem__(name, ret);
Index: PyJavaPackage.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/PyJavaPackage.java,v
retrieving revision 2.1
diff -c -r2.1 PyJavaPackage.java
*** PyJavaPackage.java	1999/05/17 19:59:39	2.1
--- PyJavaPackage.java	2000/10/19 18:36:50
***************
*** 190,198 ****
              return ret;
          Class c;
          if (__name__.length()>0)
!             c = Py.findClass(__name__+'.'+name);
          else
!             c = Py.findClass(name);
  
          if (c != null) {
              ret = PyJavaClass.lookup(c);
--- 190,198 ----
              return ret;
          Class c;
          if (__name__.length()>0)
!             c = Py.findClassEx(__name__+'.'+name);
          else
!             c = Py.findClassEx(name);
  
          if (c != null) {
              ret = PyJavaClass.lookup(c);
Index: imp.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/imp.java,v
retrieving revision 2.25
diff -c -r2.25 imp.java
*** imp.java	2000/10/18 14:00:49	2.25
--- imp.java	2000/10/19 18:36:53
***************
*** 191,197 ****
              return Py.java2py(Py.getSystemState());
          String mod = PySystemState.getBuiltin(name);
          if (mod != null) {
!             Class c = Py.findClass(mod);
              if (c != null) {
                  try {
                      return createFromClass(name, c);
--- 191,197 ----
              return Py.java2py(Py.getSystemState());
          String mod = PySystemState.getBuiltin(name);
          if (mod != null) {
!             Class c = Py.findClassEx(mod);
              if (c != null) {
                  try {
                      return createFromClass(name, c);
***************
*** 210,216 ****
          if (Py.frozenPackage != null) {
              modName = Py.frozenPackage+"."+modName;
          }
!         return Py.findClass(modName+"$_PyInner");
      }
  
      private static PyObject loadPrecompiled(String name, String modName,
--- 210,216 ----
          if (Py.frozenPackage != null) {
              modName = Py.frozenPackage+"."+modName;
          }
!         return Py.findClassEx(modName+"$_PyInner");
      }
  
      private static PyObject loadPrecompiled(String name, String modName,
***************
*** 352,358 ****
          if (ret != null) return ret;
  
          if (Py.frozen) {
!             Class c = Py.findClass(name);
              if (c != null) return createFromClass(name, c);
          }
  
--- 352,358 ----
          if (ret != null) return ret;
  
          if (Py.frozen) {
!             Class c = Py.findClassEx(name);
              if (c != null) return createFromClass(name, c);
          }
  

Re: [Jython-dev] Class.forName bug The return.

[Samuele P. <pe...@in...>]

Hi.

[Finn]
>I have not fixed the Tools/mkjava.py or Tools/Freezer.py. Neither of
>these scripts have been working for a long time due to other changes in
>the constructor of JavaMaker.

no problem for that, in my mail I was really referring to jythonc
and its use of AdapterMaker in compile.py.

regards, Samuele.

Re: [Jython-dev] Class.forName bug The return.

[<bc...@wo...>]

[Samuele ]

>A simple note:
>
>also AdapterMaker should not use Class.forName.

I'll fix this.

>A question:
>
>After these fixings is the name taking constructor for ProxyMaker still 
>necessary?

The ctor still takes a name string, but now the name only specify the
name of the class we are building.

>and it's Class.forName based build method?

I think this removes the last of the bad forName's.

I have not fixed the Tools/mkjava.py or Tools/Freezer.py. Neither of
these scripts have been working for a long time due to other changes in
the constructor of JavaMaker.

regards,
finn

Re: [Jython-dev] *loading: trying to summarize (II)

[<bc...@wo...>]

On Thu, 19 Oct 2000 14:19:19 +0200 (MET DST), you wrote:

>I have subscribed to the sourceforge lists and so I have received the last mail 
>from Finn (twice).
>Have I missed something in between? 

Not from me.

>Things seem to get clearer both for semantic and possible implementation
>up to Barry inputs.
>
>...
>
>I clarified to myself that jythonc should build the list, use the new
>order dealing with imports but it is not necessary that it find out
>that a py pkg is also a java pkg :). The modified runtime support
>can deal with this.

I can make this change. Py.initProxy will then get a new "String[]
modules" argument. I'll make it a patch,, i.e I will not commit this
until we have settled the complete semantic change.

>But there is another issue: I have not yet analyzed it in depth.
>
>One can compile a set of py modules/py pkg under a user-defined
>java pkg (package jythonc option) so for example py mod x.y
>will be userpkg.x.y.
>
>If the code in interpreted mode load a java class x.JClass what should do
>at runtime the same code when compiled and put in the pkg userpkg,
>just still try to load x.JClass or userpkg.x.JClass, try both.

x.JClass IMO. The --package option to jpythonc should only be for the
generated code, not for the entire world.

>In any case something like java.lang.* should still be loaded as java.lang.*.
>Can we separate the different cases? What is the more coherent semantic that
>can be actually implemented?

regards,
finn

Re: [Jython-dev] *loading: trying to summarize (II)

[<ba...@wo...>]

>>>>> "SP" == Samuele Pedroni <pe...@in...> writes:

    SP> Things seem to get clearer both for semantic and possible
    SP> implementation up to Barry inputs.

I've got the raw old archives, but I'll need to make a SF admin
request to get them merged in.

I'll get to answering the meat of this thread as soon as possible, but
there are a couple of other things I need to clear off my plate first.

-Barry

Re: [Jython-dev] *loading: trying to summarize (II)

[Samuele P. <pe...@in...>]

Sorry, I read Finn mail too quickly.

I think the point is clear but better to clarify:

[Finn]
> Right, I think I understand now. The list of module would then look
> like:
> 
>   module_list = ['x', 'x.y']
> 
> and the code that look for java packages would then look like this:
> 
>   if module_list != None:
>      if modName in module_list:
>          # here modName is not a java package, so there
>          # is no need to try an expensive forName(modName)
>          # across the network.
>          return null
>   // do normal java package load here.

My idea for the import logic is as follows:

 if modName in module_list:
   load compiled modName class
 else:
   do sys.path py loading unless sys.path is empty
   do java package load here (both classpath and sys.path)
   
regards, Samuele.

Re: [Jython-dev] *loading: trying to summarize (II)

[Samuele P. <pe...@in...>]

Hi.

I have subscribed to the sourceforge lists and so I have received the last mail 
from Finn (twice).
Have I missed something in between? at the moment there is no
archive access.

Things seem to get clearer both for semantic and possible implementation
up to Barry inputs.

[Finn]
> Right, I think I understand now. The list of module would then look
> like:
> 
>   module_list = ['x', 'x.y']
> 
> and the code that look for java packages would then look like this:
> 
>   if module_list != None:
>      if modName in module_list:
>          # here modName is not a java package, so there
>          # is no need to try an expensive forName(modName)
>          # across the network.
>          return null
>   // do normal java package load here.

Yes, that is what I meant and it should solve the performance problem.

As I wrote java package load should deal also with sys.path java loading
(semantic: virtually sys.path is appended to CLASSPATH)
and main "problem" is to handle sys.path more dynamic nature.

I clarified to myself that jythonc should build the list, use the new
order dealing with imports but it is not necessary that it find out
that a py pkg is also a java pkg :). The modified runtime support
can deal with this.

But there is another issue: I have not yet analyzed it in depth.

One can compile a set of py modules/py pkg under a user-defined
java pkg (package jythonc option) so for example py mod x.y
will be userpkg.x.y.

If the code in interpreted mode load a java class x.JClass what should do
at runtime the same code when compiled and put in the pkg userpkg,
just still try to load x.JClass or userpkg.x.JClass, try both.
In any case something like java.lang.* should still be loaded as java.lang.*.
Can we separate the different cases? What is the more coherent semantic that
can be actually implemented?

regards, Samuele

PS: as I wrote, all the compiler.*Maker should not use Class.forName.
I noticed that AdpterMaker is also used by jythonc so we should pay attention.

Re: [Jython-dev] *loading: trying to summarize: clarifications (ii)

[<bc...@wo...>]

[Samuele]

>[Finn]
>> >> >sys.path (py) > java pkgs > sys.path (java) maybe is better ?
>> >> >if P is py pkg, import P will load the py pkg P independently of
>> >classpath...
>> >> >One will be still able to access the shadowed java pkg through the py
>> >pkg.
>> >>
>> >> Yes, I think this would be better.
>> >
>> >Fine, but as I wrote in a succesive mail, JimH put this precedence in place
>> >to avoid possible performance penalties
>> >by lookup in "applets" case: ...
>> 
>> Jythonc already goes through great pains to detect which of the name
>> spaces imported is actually java packages. The list of packages is then
>> initialized in Py.initProperties. Could we define that this explicit
>> list of identified java packages should be searched before all the
>> others?
>
>Yes but the performance problem is with sys.path(py) not the java pkgs,
>for them we already collect static information in PackageManager
>also based on the list constructed by jythonc.
>
>What we need (my opinion): we need an analougous list for the py pkgs:
>so given a pkg name -in the import code- we can avoid to lookup for name x.y as
>a compiled py pkg/module when it is in fact a java pkg.
>We cannot just use the java pkg information because with the new order
>py pkg can hide them and we want import to deliver a py pkg if it's the case.

Right, I think I understand now. The list of module would then look
like:

  module_list = ['x', 'x.y']

and the code that look for java packages would then look like this:

  if module_list != None:
     if modName in module_list:
         # here modName is not a java package, so there
         # is no need to try an expensive forName(modName)
         # across the network.
         return null
  // do normal java package load here.

>This list can also be a real mapping py pkg -> java class.

You've lost me here. Are you saying that the module list should be like
this:

  { 'x'   : 'x.__init__', 
    'x.y' : 'x.y' }

?
 
>This would make thing clearer and enable some interesting but clean
>features. For example if your compiled code run not under a sec. mgr
>you can change the value of sys.path and load also code from there.

regards,
finn