From: Steve Y. <ste...@gm...> - 2014-05-01 01:16:11
|
This all makes sense. And I understand now, after days in this code, exactly how it's *supposed *to work. I suspect that with sufficient determination I could even make it work. However, it's likely to break after every few Jython releases. It's an approach that spends too much time fighting Python. I just wish it hadn't worked 14 years ago, so I'd not have gone so far down this road. > PS Might be easier to move your code around. Heh. Yep! Scripts, ahoy. Thanks so much for your help. I still think it's cool that Python lets you wrangle your way out of this if you're persistent enough. -steve On Wed, Apr 30, 2014 at 3:08 PM, Jim Baker <jim...@py...> wrote: > Steve, > > Hmmm, this is an interesting challenge. > > So it looks like the real issue seems to be PyJavaPackage does not allow > for modifying its __dict__; it’s a bit convoluted but > PyJavaPackage#__setattr__ ultimately tries to use > PyObject#object___setattr__, which then tries calling fastGetDict - but > for types that are not derived - that is, something like PyObjectDerived, > this returns null. This means, *no modification allowed*. > > That’s why you cannot do the following: > > >>> x = object() > >>> x.foo = 42 > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > AttributeError: 'object' object has no attribute 'foo' > > But this also is why you cannot do this in a Java package namespace, which > seems potentially useful: > > >>> from java import io as jio > >>> jio.foo = 42 > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > AttributeError: 'javapackage' object has no attribute 'foo' > > In your case, you want to make this sort of modification to javapackageinstances. I would suggest looking at > gderived.py and see if you can use it to produce a PyJavaPackageDerivedclass. You will also need to override > addPackage in this derived class. > > The package manager is similarly convoluted; you get the top level package > manager with sys.packageManager or from a specific Java package with > __mgr__. In both cases, it’s writeable. It’s lost to the mists of time > why this change to allow mutation was made about 14 years ago, but > presumably it solved someone’s problem, maybe like yours. > > The package manager in turn has this attribute topLevelPackage; > presumably you can set this to an instance of PyJavaPackageDerived. > > Assuming you’re still following along ;), we can now connect this back to > how packages are imported. The imp module in import_first and import_nextuse > JavaImportHelper.tryAddPackage, which in turns uses > PySystemState.add_package, which calls PackageManager.makeJavaPackage, > and the loop so to speak is complete. What this means is that you can use > PEP 302 to manage the import process, deferring to existing import logic; > but allow for arbitrary modules to be imported into the Java namespace. > Hopefully :) > > I’m reminded of this quote: > > “There is a building. Inside this building there is a level where no > elevator can go, and no stair can reach. This level is filled with doors. > These doors lead to many places. Hidden places. But one door is special. > One door leads to the source.” - The Keymaker, *The Matrix Reloaded* > > Hope this helps! > > PS Might be easier to move your code around. > > – Jim > > > On April 30, 2014 at 2:09:21 AM, Steve Yegge (ste...@gm...) > wrote: > > I should clarify that my importer is currently in python, like pycimport. > It turns out pycimport only handles .pyc files, not .py or $py.class > files, at least in my 2.5.4 version of Jython. So it's exactly what I need > -- working code for producing a working module directly from a file, but it > only handles that one case. > > I'm looking for similar code to handle the five other cases: .py modules, > Java modules, Java packages, (optionally) $py.class files, and Python > packages. > > I think that imp.load_source might handle .py files for me, but I won't be > able to test it until I can properly populate a package module. I guess > maybe I'll just try to build the package dict by hand, recursively, and > hope I get it right? > > > On Wed, Apr 30, 2014 at 12:53 AM, Steve Yegge <ste...@gm...>wrote: > >> Thanks Jim. This is very helpful. >> >> I've made substantial progress on my meta-importer. However I'm >> beginning to run into territory that I'd guess is unnecessarily cloning >> built-in functionality. >> >> For instance, once I've determined that an import refers to a package, is >> there a function I can call to populate an imp.new_module() object with all >> the appropriate attributes, including scanning the directory for modules >> and sub-packages? Or do I have to tediously clone the logic scattered >> around in org.python.modules.imp.findFromSource, >> org.python.modules.imp.loadFromSource and other locations? If so, how will >> I know when I'm finished, and my package module is "whole"? Is the only >> documentation the code in the Python 2.5.2 distribution's import.c / >> load_package? >> >> I'm excited that PEP 302 provides a mechanism for providing my own >> importer. But it seems like a potentially vast amount of fragile work to >> implement the semantics I'm looking for. I don't want to import a new kind >> of virtual module. I just want to import regular modules -- including >> Java, py and pyc modules -- from some extra locations that are normally >> hidden. So ideally my meta-importer would delegate most of its work, once >> it has done the basic checking to say "yes, this file or directory is a >> module". >> >> Thanks for any advice or pointers, >> >> -steve >> >> >> >> >> >> On Tue, Apr 29, 2014 at 3:50 PM, Jim Baker <jim...@py...> wrote: >> >>> Steve, >>> >>> The advice in the Jython book is sound and describes default Python >>> behavior. However, Jython 2.5 implements standard Python 2.5 import >>> semantics, which should also give you the flexibility to support the old, >>> nonstandard behavior. >>> >>> Specifically, I would suggest taking a look at import hook support, as >>> described in PEP 302 <http://legacy.python.org/dev/peps/pep-0302/>, in >>> particular the use of meta importers on sys.meta_path. Basically you >>> can intercept both finding a module path (so you can co-mingle as desired) >>> and any subsequent loading. Such support is the basis of zipfile imports. >>> >>> PEP 302 is not exactly clearly written. I would also take a look at code >>> that manipulates sys.meta_path. One module in Jython’s stdlib that >>> might give you ideas is pycimport. This module is unfortunately lacking >>> much in the way of docs/comments, but it’s pretty concise in what it does, >>> supporting both compiled .pyc files and source .py files side-by-side >>> (along with the more usual $py.class files), which is sort of like your >>> problem. >>> >>> – Jim >>> >>> On April 29, 2014 at 1:05:09 PM, Steve Yegge (ste...@gm...) >>> wrote: >>> >>> I have an old application that I'm dusting off. It contains about >>> 1600 Java files and 1400 Jython (.py) files. The last time I made any >>> major changes to it was roughly 2003. >>> >>> The Jython modules coexist with the Java files -- sometimes in the same >>> directory as Java files, and sometimes in their own subdirectories. I >>> import these Python modules by calling a PythonInterpreter's exec function >>> from within my Java process. >>> >>> Here is a reduction that illustrates a structure similar to that of my >>> app: >>> >>> /apps/myapp/ >>> /apps/myapp/__init__.py >>> /apps/myapp/foo/ >>> /apps/myapp/foo/__init__.py >>> /apps/myapp/foo/A.java >>> /apps/myapp/foo/b.py >>> /apps/myapp/bar/__init__.py >>> /apps/myapp/bar/c.py >>> /apps/myapp/baz/D.java >>> >>> In this layout I have: >>> >>> * java packages myapp.foo and myapp.bar >>> * java classes myapp.foo.A and myapp.bar.D >>> * python packages myapp, myapp.foo, and myapp.bar >>> * python modules myapp.foo.b and myapp.bar.c >>> >>> I've upgraded to Jython 2.5.2; I think before I may have been using 2.2, >>> but it's tricky to reproduce the old setup. >>> >>> I can no longer import any Jython modules from my tree. If, for >>> instance, I try this: >>> >>> myPythonInterpreterInstance.exec("from myapp.foo import b"); >>> >>> I get: "ImportError: No module named foo". Similarly for myapp.bar.c. >>> In an interactive embedded repl in my app, I can do: >>> >>> >>> dir(myapp) >>> ['foo', 'baz'] >>> >>> But foo is in fact some sort of mirage: >>> >>> >> dir(myapp.foo) >>> Traceback (most recent call last): >>> File "<stdin>", line 1, in <module> >>> AttributeError: 'javapackage' object has no attribute 'foo' >>> >>> Needless to say I have eagerly scoured the web for potential leads. >>> Unfortunately the only lead I've found seems rather dire; it's from the >>> Definitive Jython book in Chapter 8: >>> >>> *Developers coming from Java will often make the mistake of modeling >>> their Jython package structure the same way that they model Java >>> packages. Do not do this. The reverse url convention of Java is a great, we >>> would even say a brilliant convention for Java. It works very well indeed >>> in the world of Java where these namespaces are merged. In the Python world >>> however, where modules and packages display the winner-take-all semantic, >>> this is a disastrous way to organize your code.* >>> >>> *If you adopt this style for Python, say you are coming from ‘acme.com >>> <http://acme.com/>,’ you would set up a package structure like ‘com.acme.’ >>> If you try to use a library from your vendor xyz that is set up as >>> ‘com.xyz,’ then the first of these on your path will take the ‘com’ >>> namespace, and you will not be able to see the other set of packages.* >>> Is this my problem? Was this particular merging behavior >>> unimplemented in Jython 2.2, and that's why the reverse-hierarchical >>> structure worked for me before? >>> >>> To test whether this might indeed be my problem, I modified my app >>> structure as follows: >>> >>> /apps/__init__.py >>> /apps/test/__init__.py >>> /apps/test/b.py >>> /apps/myapp/ >>> /apps/myapp/__init__.py >>> /apps/myapp/foo/ >>> /apps/myapp/foo/__init__.py >>> /apps/myapp/foo/A.java >>> /apps/myapp/foo/b.py >>> /apps/myapp/bar/__init__.py >>> /apps/myapp/bar/c.py >>> /apps/myapp/baz/D.java >>> >>> That is, I created a new directory peer of myapp, and copied b.py there. >>> I successfully imported b.py from within myapp: >>> >>> >>> import test >>> import py >>> >>> dir(py) >>> ['__doc__', '__file__', '__loader__', '__name__', '__path__'] >>> >>> from(py) import b >>> >>> dir(b) >>> ... >>> >>> I haven't tried it from the Java side using the PythonInterpreter, as >>> it's tricky to set up. But I suspect it would work, as my repl is simply >>> delegating to it anyway. >>> >>> My hypothesis, from the observed behavior and from the warning in >>> Definitive Jython, is that my Java packages are effectively hiding the >>> Python packages and modules living in the same directory structure. >>> >>> Is that accurate? If so, is there any cure? Relocating 1300 Jython >>> modules to new homes will be nontrivial -- especially as they are >>> referenced in hundreds of locations in my code, and also in over 35,000 >>> application data files. I am greatly hopeful that someone will produce a >>> nifty metaprogramming trick that lets me keep the current layout. That, or >>> (even better) perhaps I am simply dead wrong about the root cause. >>> >>> (Note: I tried a heavyweight workaround, changing the exec() to an >>> execfile() on the absolute path to the module. This works on most systems >>> I tried. But I fear that lacking the normal caching of the import >>> mechanism(?), it will have unacceptable runtime performance at scale.) >>> >>> -steve >>> ------------------------------------------------------------------------------ >>> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE >>> Instantly run your Selenium tests across 300+ browser/OS combos. Get >>> unparalleled scalability from the best Selenium testing platform >>> available. >>> Simple to use. Nothing to install. Get started now for free." >>> >>> http://p.sf.net/sfu/SauceLabs_______________________________________________ >>> Jython-users mailing list >>> Jyt...@li... >>> https://lists.sourceforge.net/lists/listinfo/jython-users >>> >>> >> > |