From: Alan F. <Ala...@sa...> - 2007-02-28 19:18:30
|
(Cross-posting to jython-dev since it has actual development content!) OK, I figured it out. First I turned on debug messaging in the registry = file to see if that would provide any clues. Here is the trace when the = import is successful: >>> from SAS.DIServer import MDOContext <single-top>: MDOContext=3D import: trying source C:\jython2.2b1\SAS import: trying precompiled with no sourceC:\jython2.2b1\SAS$py.class import: trying source C:\jython2.2b1\SAS import: trying precompiled with no sourceC:\jython2.2b1\SAS$py.class import: trying source C:\jython2.2b1\Lib\SAS import: trying precompiled with no sourceC:\jython2.2b1\Lib\SAS$py.class #This is the point at which the import succeeds... import: trying precompiled C:\SASJythonPackage\SAS\__init__$py.class import: 'SAS' as C:\SASJythonPackage\SAS\__init__.py And here is the trace when the import is not successful: >>> from SAS.DIServer import MDOContext <single-top>: MDOContext=3D import: trying source C:\jython2.2b1\SAS import: trying precompiled with no sourceC:\jython2.2b1\SAS$py.class import: trying source C:\jython2.2b1\SAS import: trying precompiled with no sourceC:\jython2.2b1\SAS$py.class import: trying source C:\jython2.2b1\Lib\SAS import: trying precompiled with no sourceC:\jython2.2b1\Lib\SAS$py.class #This is the point where the trace deviates from the successful trace... import: trying SAS in packagemanager for path None import: java package as 'C:\SASJythonPackage\SAS' import: 'SAS' as java package import: 'SAS' as java package import: trying SAS.DIServer as java class in syspath loader Traceback (innermost last): File "<console>", line 1, in ? ImportError: no module named DIServer So I found the "import: trying <something> in packagemanager for path = <something>" message in org.python.core.JavaImporter.find_module(). Then = I stepped through the code while the import was happening. Here is the = call stack:=20 org.python.core.JavaImporter.find_module() org.python.core.PackageManager.lookupName() org.python.core.PyJavaPackage.__findattr__() org.python.core.SysPackageManager.packageExists() org.python.core.PathPackageManager.packageExists() In org.python.core.PathPackageManager.packageExists() a = org.python.core.PackageExistsFileFilter object is created which looks = through the files in the directory to see if the directory is a Java or = Python package. Here is where it gets interesting. At some point in the = past, I was using jythonc to compile some classes and one of these = classes was still in the directory. It is named = "message$_PyInner.class". I don't have the original source code anymore, = so I am not exactly sure what created this class but this is the root = cause of my import problem. The = org.python.core.PackageExistsFileFilter.accept() method tests for files = that end in ".py" and "$py.class". If it finds any files with this = extension, then the directory is treated as a Python package. If it = finds a file that ends in ".class", then the directory is treated as a = Java package. I added a test to the = org.python.core.PackageExistsFileFilter.accept() method for = "$_PyInner.class", and then the import succeeded. This also explains why this wasn't a name clash. The $_PyInner.class existed = in the directory no matter what the name. Here is the updated accept = method: public boolean accept(File dir, String name) { if (name.endsWith(".py") || name.endsWith("$py.class") || = name.endsWith("$_PyInner.class")) { this.python =3D true; } else if (name.endsWith(".class")) { this.java =3D true; } return false; } I can enter a bug in the bugtracker, but I do not know if there are any = other file extensions generated by jythonc that might cause this = problem. Anybody out there know? Thanks for the suggestions. Alan |
From: Oti <oh...@gm...> - 2007-02-28 20:27:40
|
Alan, my compliments - great analysis ! Unfortunately I do not know enough about jythonc to give a list of possible class file endings. I'll check if I can find the $_PyInner.class somewhere in the code and deduce the rest from it. best wishes, Oti. On 2/28/07, Alan Field <Ala...@sa...> wrote: > (Cross-posting to jython-dev since it has actual development content!) > > OK, I figured it out. First I turned on debug messaging in the registry file to see if that would provide any clues. Here is the trace when the import is successful: > > >>> from SAS.DIServer import MDOContext > <single-top>: MDOContext= > import: trying source C:\jython2.2b1\SAS > import: trying precompiled with no sourceC:\jython2.2b1\SAS$py.class > import: trying source C:\jython2.2b1\SAS > import: trying precompiled with no sourceC:\jython2.2b1\SAS$py.class > import: trying source C:\jython2.2b1\Lib\SAS > import: trying precompiled with no sourceC:\jython2.2b1\Lib\SAS$py.class > #This is the point at which the import succeeds... > import: trying precompiled C:\SASJythonPackage\SAS\__init__$py.class > import: 'SAS' as C:\SASJythonPackage\SAS\__init__.py > > And here is the trace when the import is not successful: > >>> from SAS.DIServer import MDOContext > <single-top>: MDOContext= > import: trying source C:\jython2.2b1\SAS > import: trying precompiled with no sourceC:\jython2.2b1\SAS$py.class > import: trying source C:\jython2.2b1\SAS > import: trying precompiled with no sourceC:\jython2.2b1\SAS$py.class > import: trying source C:\jython2.2b1\Lib\SAS > import: trying precompiled with no sourceC:\jython2.2b1\Lib\SAS$py.class > #This is the point where the trace deviates from the successful trace... > import: trying SAS in packagemanager for path None > import: java package as 'C:\SASJythonPackage\SAS' > import: 'SAS' as java package > import: 'SAS' as java package > import: trying SAS.DIServer as java class in syspath loader > Traceback (innermost last): > File "<console>", line 1, in ? > ImportError: no module named DIServer > > So I found the "import: trying <something> in packagemanager for path <something>" message in org.python.core.JavaImporter.find_module(). Then I stepped through the code while the import was happening. Here is the call stack: > > org.python.core.JavaImporter.find_module() > org.python.core.PackageManager.lookupName() > org.python.core.PyJavaPackage.__findattr__() > org.python.core.SysPackageManager.packageExists() > org.python.core.PathPackageManager.packageExists() > > In org.python.core.PathPackageManager.packageExists() a org.python.core.PackageExistsFileFilter object is created which looks through the files in the directory to see if the directory is a Java or Python package. Here is where it gets interesting. At some point in the past, I was using jythonc to compile some classes and one of these classes was still in the directory. It is named "message$_PyInner.class". I don't have the original source code anymore, so I am not exactly sure what created this class but this is the root cause of my import problem. The org.python.core.PackageExistsFileFilter.accept() method tests for files that end in ".py" and "$py.class". If it finds any files with this extension, then the directory is treated as a Python package. If it finds a file that ends in ".class", then the directory is treated as a Java package. I added a test to the org.python.core.PackageExistsFileFilter.accept() method for "$_PyInner.class", and then the import succeeded. This > also explains why this wasn't a name clash. The $_PyInner.class existed in the directory no matter what the name. Here is the updated accept method: > > public boolean accept(File dir, String name) { > if (name.endsWith(".py") || name.endsWith("$py.class") || name.endsWith("$_PyInner.class")) { > this.python = true; > } else if (name.endsWith(".class")) { > this.java = true; > } > return false; > } > > I can enter a bug in the bugtracker, but I do not know if there are any other file extensions generated by jythonc that might cause this problem. Anybody out there know? > > Thanks for the suggestions. > Alan > |
From: Alan F. <Ala...@sa...> - 2007-02-28 20:46:03
|
Oti, Thanks a lot. Finding the debug messages was really a big help. I did = not know about them until I found them in the source code, and then = checked the registry file. The messages helped to narrow down which = method was behaving incorrectly. Good luck finding the other jythonc = file extensions. I haven't had any luck so far. The more I think about this, the more I am concerned though. Basically = my import was failing because the directory was being flagged as both a = Java and Python package. Adding the test for "$_PyInner.class" will fix = my situation, but it seems like the import will still fail if I have a = Java class in the default package in the same directory as a Python = "__init__.py" file. I'll try an experiment to see if this is true, but I = think this is a scenario that should work. Thanks, Alan -----Original Message----- From: Oti [mailto:oh...@gm...]=20 Sent: Wednesday, February 28, 2007 3:27 PM To: Alan Field Cc: Charlie Groves; jython users; Jython-Dev Subject: Re: [Jython-users] Problems with import and Jython 2.2 Beta1... Alan, my compliments - great analysis ! Unfortunately I do not know enough about jythonc to give a list of = possible class file endings. I'll check if I can find the = $_PyInner.class somewhere in the code and deduce the rest from it. best wishes, Oti. |
From: Oti <oh...@gm...> - 2007-02-28 21:43:10
|
Alan, I found 2 occurrences in org.python.core.PrecompiledImporter org.python.core.Py and the comments around there strongly indicate that these are precompiled classes. On the other hand, there is code in Tools/jythonc/PythonModule.py where an inner class named _PyInner is added. Since inner classes are separated by a $ sign from the surrounding class, something ending in $_PyInner.class is very likely to result. See also: http://sourceforge.net/tracker/index.php?func=detail&aid=480373&group_id=12867&atid=112867 It is almost impossible for me to detect other secret class name endings. There is a comment in PathPackageManager saying /* * Figure out if we have a directory a mixture of python and * java or just an empty directory (which means Java) or a * directory with only Python source (which means Python). */ So, from a high level view, your approach of preventing $_PyInner.class from indicating a mixture (leading to java) seems right. Regarding your concerns about really mixing java packages and python modules, I believe we should not get too paranoid about that. IMHO common sense will prevent people from messing with such a setup. I suggest I run the tests with and without your patch, and if I do not encounter strange effects, it'll get in. Thanks again! Oti. On 2/28/07, Alan Field <Ala...@sa...> wrote: > Oti, > > Thanks a lot. Finding the debug messages was really a big help. I did not know about them until I found them in the source code, and then checked the registry file. The messages helped to narrow down which method was behaving incorrectly. Good luck finding the other jythonc file extensions. I haven't had any luck so far. > > The more I think about this, the more I am concerned though. Basically my import was failing because the directory was being flagged as both a Java and Python package. Adding the test for "$_PyInner.class" will fix my situation, but it seems like the import will still fail if I have a Java class in the default package in the same directory as a Python "__init__.py" file. I'll try an experiment to see if this is true, but I think this is a scenario that should work. > > Thanks, > Alan > > -----Original Message----- > From: Oti [mailto:oh...@gm...] > Sent: Wednesday, February 28, 2007 3:27 PM > To: Alan Field > Cc: Charlie Groves; jython users; Jython-Dev > Subject: Re: [Jython-users] Problems with import and Jython 2.2 Beta1... > > Alan, > > my compliments - great analysis ! > > Unfortunately I do not know enough about jythonc to give a list of possible class file endings. I'll check if I can find the $_PyInner.class somewhere in the code and deduce the rest from it. > > best wishes, > Oti. > |
From: Oti <oh...@gm...> - 2007-03-02 06:56:42
|
Alan, at the moment I get one difference in bugtests/test392. I'm still on it. Oti. On 2/28/07, Oti <oh...@gm...> wrote: > Alan, > > I found 2 occurrences in > org.python.core.PrecompiledImporter > org.python.core.Py > and the comments around there strongly indicate that these are > precompiled classes. > > On the other hand, there is code in > Tools/jythonc/PythonModule.py > where an inner class named _PyInner is added. > Since inner classes are separated by a $ sign from the surrounding > class, something ending in $_PyInner.class is very likely to result. > > See also: > http://sourceforge.net/tracker/index.php?func=detail&aid=480373&group_id=12867&atid=112867 > > It is almost impossible for me to detect other secret class name endings. > > There is a comment in PathPackageManager saying > /* > * Figure out if we have a directory a mixture of python and > * java or just an empty directory (which means Java) or a > * directory with only Python source (which means Python). > */ > > So, from a high level view, your approach of preventing > $_PyInner.class from indicating a mixture (leading to java) seems > right. > > Regarding your concerns about really mixing java packages and python > modules, I believe we should not get too paranoid about that. IMHO > common sense will prevent people from messing with such a setup. > > I suggest I run the tests with and without your patch, and if I do not > encounter strange effects, it'll get in. > > Thanks again! > Oti. > > > > On 2/28/07, Alan Field <Ala...@sa...> wrote: > > Oti, > > > > Thanks a lot. Finding the debug messages was really a big help. I did not know about them until I found them in the source code, and then checked the registry file. The messages helped to narrow down which method was behaving incorrectly. Good luck finding the other jythonc file extensions. I haven't had any luck so far. > > > > The more I think about this, the more I am concerned though. Basically my import was failing because the directory was being flagged as both a Java and Python package. Adding the test for "$_PyInner.class" will fix my situation, but it seems like the import will still fail if I have a Java class in the default package in the same directory as a Python "__init__.py" file. I'll try an experiment to see if this is true, but I think this is a scenario that should work. > > > > Thanks, > > Alan > > > > -----Original Message----- > > From: Oti [mailto:oh...@gm...] > > Sent: Wednesday, February 28, 2007 3:27 PM > > To: Alan Field > > Cc: Charlie Groves; jython users; Jython-Dev > > Subject: Re: [Jython-users] Problems with import and Jython 2.2 Beta1... > > > > Alan, > > > > my compliments - great analysis ! > > > > Unfortunately I do not know enough about jythonc to give a list of possible class file endings. I'll check if I can find the $_PyInner.class somewhere in the code and deduce the rest from it. > > > > best wishes, > > Oti. > > > |