Steve,

The advice in the Jython book is sound and describes default Python behavior. However, Jython 2.5 implements standard Python 2.5 import semantics, which should also give you the flexibility to support the old, nonstandard behavior.

Specifically, I would suggest taking a look at import hook support, as described in PEP 302, in particular the use of meta importers on sys.meta_path. Basically you can intercept both finding a module path (so you can co-mingle as desired) and any subsequent loading. Such support is the basis of zipfile imports.

PEP 302 is not exactly clearly written. I would also take a look at code that manipulates sys.meta_path. One module in Jython’s stdlib that might give you ideas is pycimport. This module is unfortunately lacking much in the way of docs/comments, but it’s pretty concise in what it does, supporting both compiled .pyc files and source .py files side-by-side (along with the more usual $py.class files), which is sort of like your problem.

– Jim


On April 29, 2014 at 1:05:09 PM, Steve Yegge (steve.yegge@gmail.com) wrote:

I have an old application that I'm dusting off.  It contains about 1600 Java files and 1400 Jython (.py) files.  The last time I made any major changes to it was roughly 2003.

The Jython modules coexist with the Java files -- sometimes in the same directory as Java files, and sometimes in their own subdirectories.  I import these Python modules by calling a PythonInterpreter's exec function from within my Java process.

Here is a reduction that illustrates a structure similar to that of my app:

/apps/myapp/
/apps/myapp/__init__.py
/apps/myapp/foo/
/apps/myapp/foo/__init__.py
/apps/myapp/foo/A.java
/apps/myapp/foo/b.py
/apps/myapp/bar/__init__.py
/apps/myapp/bar/c.py
/apps/myapp/baz/D.java

In this layout I have:

  * java packages myapp.foo and myapp.bar
  * java classes myapp.foo.A and myapp.bar.D
  * python packages myapp, myapp.foo, and myapp.bar
  * python modules myapp.foo.b and myapp.bar.c

I've upgraded to Jython 2.5.2; I think before I may have been using 2.2, but it's tricky to reproduce the old setup.

I can no longer import any Jython modules from my tree.  If, for instance, I try this:

myPythonInterpreterInstance.exec("from myapp.foo import b");

I get:  "ImportError: No module named foo".  Similarly for myapp.bar.c.  In an interactive embedded repl in my app, I can do:

>>> dir(myapp)
['foo', 'baz']

But foo is in fact some sort of mirage:

>> dir(myapp.foo)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'javapackage' object has no attribute 'foo'

Needless to say I have eagerly scoured the web for potential leads.  Unfortunately the only lead I've found seems rather dire; it's from the Definitive Jython book in Chapter 8:

Developers coming from Java will often make the mistake of modeling their Jython package structure the same way that they model Java packages. Do not do this. The reverse url convention of Java is a great, we would even say a brilliant convention for Java. It works very well indeed in the world of Java where these namespaces are merged. In the Python world however, where modules and packages display the winner-take-all semantic, this is a disastrous way to organize your code.

If you adopt this style for Python, say you are coming from ‘acme.com,’ you would set up a package structure like ‘com.acme.’ If you try to use a library from your vendor xyz that is set up as ‘com.xyz,’ then the first of these on your path will take the ‘com’ namespace, and you will not be able to see the other set of packages.

Is this my problem?  Was this particular merging behavior unimplemented in Jython 2.2, and that's why the reverse-hierarchical structure worked for me before?

To test whether this might indeed be my problem, I modified my app structure as follows:

/apps/__init__.py
/apps/test/__init__.py
/apps/test/b.py
/apps/myapp/
/apps/myapp/__init__.py
/apps/myapp/foo/
/apps/myapp/foo/__init__.py
/apps/myapp/foo/A.java
/apps/myapp/foo/b.py
/apps/myapp/bar/__init__.py
/apps/myapp/bar/c.py
/apps/myapp/baz/D.java

That is, I created a new directory peer of myapp, and copied b.py there.  I successfully imported b.py from within myapp:

>>> import test
import py
>>> dir(py)
['__doc__', '__file__', '__loader__', '__name__', '__path__']
>>> from(py) import b
>>> dir(b)
...

I haven't tried it from the Java side using the PythonInterpreter, as it's tricky to set up.  But I suspect it would work, as my repl is simply delegating to it anyway.

My hypothesis, from the observed behavior and from the warning in Definitive Jython, is that my Java packages are effectively hiding the Python packages and modules living in the same directory structure.

Is that accurate?  If so, is there any cure?  Relocating 1300 Jython modules to new homes will be nontrivial -- especially as they are referenced in hundreds of locations in my code, and also in over 35,000 application data files.  I am greatly hopeful that someone will produce a nifty metaprogramming trick that lets me keep the current layout.  That, or (even better) perhaps I am simply dead wrong about the root cause.

(Note:  I tried a heavyweight workaround, changing the exec() to an execfile() on the absolute path to the module.  This works on most systems I tried.  But I fear that lacking the normal caching of the import mechanism(?), it will have unacceptable runtime performance at scale.)

-steve
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos. Get
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs_______________________________________________
Jython-users mailing list
Jython-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jython-users