Steve,

Hmmm, this is an interesting challenge.

So it looks like the real issue seems to be PyJavaPackage does not allow for modifying its __dict__; it’s a bit convoluted but PyJavaPackage#__setattr__ ultimately tries to use PyObject#object___setattr__, which then tries calling fastGetDict - but for types that are not derived - that is, something like PyObjectDerived, this returns null. This means, no modification allowed.

That’s why you cannot do the following:

>>> x = object()
>>> x.foo = 42
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'object' object has no attribute 'foo'

But this also is why you cannot do this in a Java package namespace, which seems potentially useful:

>>> from java import io as jio
>>> jio.foo = 42
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'javapackage' object has no attribute 'foo'

In your case, you want to make this sort of modification to javapackage instances. I would suggest looking at gderived.py and see if you can use it to produce a PyJavaPackageDerived class. You will also need to override addPackage in this derived class.

The package manager is similarly convoluted; you get the top level package manager with sys.packageManager or from a specific Java package with __mgr__. In both cases, it’s writeable. It’s lost to the mists of time why this change to allow mutation was made about 14 years ago, but presumably it solved someone’s problem, maybe like yours.

The package manager in turn has this attribute topLevelPackage; presumably you can set this to an instance of PyJavaPackageDerived.

Assuming you’re still following along ;), we can now connect this back to how packages are imported. The imp module in import_first and import_next use JavaImportHelper.tryAddPackage, which in turns uses PySystemState.add_package, which calls PackageManager.makeJavaPackage, and the loop so to speak is complete. What this means is that you can use PEP 302 to manage the import process, deferring to existing import logic; but allow for arbitrary modules to be imported into the Java namespace. Hopefully :)

I’m reminded of this quote:

“There is a building. Inside this building there is a level where no elevator can go, and no stair can reach. This level is filled with doors. These doors lead to many places. Hidden places. But one door is special. One door leads to the source.” - The Keymaker, The Matrix Reloaded

Hope this helps!

PS Might be easier to move your code around.

– Jim


On April 30, 2014 at 2:09:21 AM, Steve Yegge (steve.yegge@gmail.com) wrote:

I should clarify that my importer is currently in python, like pycimport.  It turns out pycimport only handles .pyc files, not .py or $py.class files, at least in my 2.5.4 version of Jython.  So it's exactly what I need -- working code for producing a working module directly from a file, but it only handles that one case.

I'm looking for similar code to handle the five other cases:  .py modules, Java modules, Java packages, (optionally) $py.class files, and Python packages.

I think that imp.load_source might handle .py files for me, but I won't be able to test it until I can properly populate a package module.  I guess maybe I'll just try to build the package dict by hand, recursively, and hope I get it right?


On Wed, Apr 30, 2014 at 12:53 AM, Steve Yegge <steve.yegge@gmail.com> wrote:
Thanks Jim.  This is very helpful.

I've made substantial progress on my meta-importer.  However I'm beginning to run into territory that I'd guess is unnecessarily cloning built-in functionality.

For instance, once I've determined that an import refers to a package, is there a function I can call to populate an imp.new_module() object with all the appropriate attributes, including scanning the directory for modules and sub-packages?  Or do I have to tediously clone the logic scattered around in org.python.modules.imp.findFromSource, org.python.modules.imp.loadFromSource and other locations?  If so, how will I know when I'm finished, and my package module is "whole"?  Is the only documentation the code in the Python 2.5.2 distribution's import.c / load_package?

I'm excited that PEP 302 provides a mechanism for providing my own importer.  But it seems like a potentially vast amount of fragile work to implement the semantics I'm looking for.  I don't want to import a new kind of virtual module.  I just want to import regular modules -- including Java, py and pyc modules -- from some extra locations that are normally hidden.  So ideally my meta-importer would delegate most of its work, once it has done the basic checking to say "yes, this file or directory is a module".

Thanks for any advice or pointers,

-steve





On Tue, Apr 29, 2014 at 3:50 PM, Jim Baker <jim.baker@python.org> wrote:

Steve,

The advice in the Jython book is sound and describes default Python behavior. However, Jython 2.5 implements standard Python 2.5 import semantics, which should also give you the flexibility to support the old, nonstandard behavior.

Specifically, I would suggest taking a look at import hook support, as described in PEP 302, in particular the use of meta importers on sys.meta_path. Basically you can intercept both finding a module path (so you can co-mingle as desired) and any subsequent loading. Such support is the basis of zipfile imports.

PEP 302 is not exactly clearly written. I would also take a look at code that manipulates sys.meta_path. One module in Jython’s stdlib that might give you ideas is pycimport. This module is unfortunately lacking much in the way of docs/comments, but it’s pretty concise in what it does, supporting both compiled .pyc files and source .py files side-by-side (along with the more usual $py.class files), which is sort of like your problem.

– Jim


On April 29, 2014 at 1:05:09 PM, Steve Yegge (steve.yegge@gmail.com) wrote:

I have an old application that I'm dusting off.  It contains about 1600 Java files and 1400 Jython (.py) files.  The last time I made any major changes to it was roughly 2003.

The Jython modules coexist with the Java files -- sometimes in the same directory as Java files, and sometimes in their own subdirectories.  I import these Python modules by calling a PythonInterpreter's exec function from within my Java process.

Here is a reduction that illustrates a structure similar to that of my app:

/apps/myapp/
/apps/myapp/__init__.py
/apps/myapp/foo/
/apps/myapp/foo/__init__.py
/apps/myapp/foo/A.java
/apps/myapp/foo/b.py
/apps/myapp/bar/__init__.py
/apps/myapp/bar/c.py
/apps/myapp/baz/D.java

In this layout I have:

  * java packages myapp.foo and myapp.bar
  * java classes myapp.foo.A and myapp.bar.D
  * python packages myapp, myapp.foo, and myapp.bar
  * python modules myapp.foo.b and myapp.bar.c

I've upgraded to Jython 2.5.2; I think before I may have been using 2.2, but it's tricky to reproduce the old setup.

I can no longer import any Jython modules from my tree.  If, for instance, I try this:

myPythonInterpreterInstance.exec("from myapp.foo import b");

I get:  "ImportError: No module named foo".  Similarly for myapp.bar.c.  In an interactive embedded repl in my app, I can do:

>>> dir(myapp)
['foo', 'baz']

But foo is in fact some sort of mirage:

>> dir(myapp.foo)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'javapackage' object has no attribute 'foo'

Needless to say I have eagerly scoured the web for potential leads.  Unfortunately the only lead I've found seems rather dire; it's from the Definitive Jython book in Chapter 8:

Developers coming from Java will often make the mistake of modeling their Jython package structure the same way that they model Java packages. Do not do this. The reverse url convention of Java is a great, we would even say a brilliant convention for Java. It works very well indeed in the world of Java where these namespaces are merged. In the Python world however, where modules and packages display the winner-take-all semantic, this is a disastrous way to organize your code.

If you adopt this style for Python, say you are coming from ‘acme.com,’ you would set up a package structure like ‘com.acme.’ If you try to use a library from your vendor xyz that is set up as ‘com.xyz,’ then the first of these on your path will take the ‘com’ namespace, and you will not be able to see the other set of packages.

Is this my problem?  Was this particular merging behavior unimplemented in Jython 2.2, and that's why the reverse-hierarchical structure worked for me before?

To test whether this might indeed be my problem, I modified my app structure as follows:

/apps/__init__.py
/apps/test/__init__.py
/apps/test/b.py
/apps/myapp/
/apps/myapp/__init__.py
/apps/myapp/foo/
/apps/myapp/foo/__init__.py
/apps/myapp/foo/A.java
/apps/myapp/foo/b.py
/apps/myapp/bar/__init__.py
/apps/myapp/bar/c.py
/apps/myapp/baz/D.java

That is, I created a new directory peer of myapp, and copied b.py there.  I successfully imported b.py from within myapp:

>>> import test
import py
>>> dir(py)
['__doc__', '__file__', '__loader__', '__name__', '__path__']
>>> from(py) import b
>>> dir(b)
...

I haven't tried it from the Java side using the PythonInterpreter, as it's tricky to set up.  But I suspect it would work, as my repl is simply delegating to it anyway.

My hypothesis, from the observed behavior and from the warning in Definitive Jython, is that my Java packages are effectively hiding the Python packages and modules living in the same directory structure.

Is that accurate?  If so, is there any cure?  Relocating 1300 Jython modules to new homes will be nontrivial -- especially as they are referenced in hundreds of locations in my code, and also in over 35,000 application data files.  I am greatly hopeful that someone will produce a nifty metaprogramming trick that lets me keep the current layout.  That, or (even better) perhaps I am simply dead wrong about the root cause.

(Note:  I tried a heavyweight workaround, changing the exec() to an execfile() on the absolute path to the module.  This works on most systems I tried.  But I fear that lacking the normal caching of the import mechanism(?), it will have unacceptable runtime performance at scale.)

-steve
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos. Get
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs_______________________________________________
Jython-users mailing list
Jython-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jython-users