I have an old application that I'm dusting off.  It contains about 1600 Java files and 1400 Jython (.py) files.  The last time I made any major changes to it was roughly 2003.

The Jython modules coexist with the Java files -- sometimes in the same directory as Java files, and sometimes in their own subdirectories.  I import these Python modules by calling a PythonInterpreter's exec function from within my Java process.

Here is a reduction that illustrates a structure similar to that of my app:

/apps/myapp/
/apps/myapp/__init__.py
/apps/myapp/foo/
/apps/myapp/foo/__init__.py
/apps/myapp/foo/A.java
/apps/myapp/foo/b.py
/apps/myapp/bar/__init__.py
/apps/myapp/bar/c.py
/apps/myapp/baz/D.java

In this layout I have:

  * java packages myapp.foo and myapp.bar
  * java classes myapp.foo.A and myapp.bar.D
  * python packages myapp, myapp.foo, and myapp.bar
  * python modules myapp.foo.b and myapp.bar.c

I've upgraded to Jython 2.5.2; I think before I may have been using 2.2, but it's tricky to reproduce the old setup.

I can no longer import any Jython modules from my tree.  If, for instance, I try this:

myPythonInterpreterInstance.exec("from myapp.foo import b");

I get:  "ImportError: No module named foo".  Similarly for myapp.bar.c.  In an interactive embedded repl in my app, I can do:

>>> dir(myapp)
['foo', 'baz']

But foo is in fact some sort of mirage:

>> dir(myapp.foo)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'javapackage' object has no attribute 'foo'

Needless to say I have eagerly scoured the web for potential leads.  Unfortunately the only lead I've found seems rather dire; it's from the Definitive Jython book in Chapter 8:

Developers coming from Java will often make the mistake of modeling their Jython package structure the same way that they model Java packages. Do not do this. The reverse url convention of Java is a great, we would even say a brilliant convention for Java. It works very well indeed in the world of Java where these namespaces are merged. In the Python world however, where modules and packages display the winner-take-all semantic, this is a disastrous way to organize your code.

If you adopt this style for Python, say you are coming from ‘acme.com,’ you would set up a package structure like ‘com.acme.’ If you try to use a library from your vendor xyz that is set up as ‘com.xyz,’ then the first of these on your path will take the ‘com’ namespace, and you will not be able to see the other set of packages.

Is this my problem?  Was this particular merging behavior unimplemented in Jython 2.2, and that's why the reverse-hierarchical structure worked for me before?

To test whether this might indeed be my problem, I modified my app structure as follows:

/apps/__init__.py
/apps/test/__init__.py
/apps/test/b.py
/apps/myapp/
/apps/myapp/__init__.py
/apps/myapp/foo/
/apps/myapp/foo/__init__.py
/apps/myapp/foo/A.java
/apps/myapp/foo/b.py
/apps/myapp/bar/__init__.py
/apps/myapp/bar/c.py
/apps/myapp/baz/D.java

That is, I created a new directory peer of myapp, and copied b.py there.  I successfully imported b.py from within myapp:

>>> import test
import py
>>> dir(py)
['__doc__', '__file__', '__loader__', '__name__', '__path__']
>>> from(py) import b
>>> dir(b)
...

I haven't tried it from the Java side using the PythonInterpreter, as it's tricky to set up.  But I suspect it would work, as my repl is simply delegating to it anyway.

My hypothesis, from the observed behavior and from the warning in Definitive Jython, is that my Java packages are effectively hiding the Python packages and modules living in the same directory structure.

Is that accurate?  If so, is there any cure?  Relocating 1300 Jython modules to new homes will be nontrivial -- especially as they are referenced in hundreds of locations in my code, and also in over 35,000 application data files.  I am greatly hopeful that someone will produce a nifty metaprogramming trick that lets me keep the current layout.  That, or (even better) perhaps I am simply dead wrong about the root cause.

(Note:  I tried a heavyweight workaround, changing the exec() to an execfile() on the absolute path to the module.  This works on most systems I tried.  But I fear that lacking the normal caching of the import mechanism(?), it will have unacceptable runtime performance at scale.)

-steve