From: Charlie G. <cha...@gm...> - 2009-11-03 19:38:43
|
On Nov 2, 2009, at 3:42, "Jan Wedel" <Jan...@et...> wrote: > Hi! > >> If you just want to use arbitrary Python modules in Jython without >> dynamic compilation and don't care about exposing them as Java >> classes, clamp isn't adding anything. However, one of the things I >> added to Jython for clamp is the ability to specify where to store >> the >> bytecode Jython generates and that stored bytecode should be used >> instead of always dynamically generating it. That should allow you >> to >> use 3rd party libs in a restricted environment. > > Do you mean Java byte code or Python byte code? If Python actually > generates Java byte code, where can I find it? Do you know any > documentation by chance, that explains how to generate Java class > files > or (in an optimal way) create a self-contained jar file from Python > code? It's java byte code. Whenever you import a module with jython, it spits out a $py.class file corresponding to the imported module in the same directory. You can use the compileall module included with jython to compile a whole directory tree and control where it's output. After compiling everything, you no longer need the .py files. I think there are a couple guides on the wiki for doing this. However jython still generates java bytecode for proxy classes at runtime, even if the modules containing the proxy classes are precompiled. This is what the proxy generation branch takes care of. >> I'm a little worried about your desire for "native speed". Jython is >> a decently fast implementation of Python, but it's not going to be as >> fast as plain Java code. I don't have a good feel for what the ratio >> is of Java performance to Jython performance these days, especially >> for VMs other than Sun's, so I can't say how much slower Jython will >> be. What level of performance do you need? Maybe someone else here >> can give you an idea of how Jython will do. > > I've done some performance tests on my Python Interpreter. It runs > 200-300 times slower that the same program coded in "native" Java. > Hmm, > I think I'm still a bit confused about what Jython does. > > Is Jython "only" a Java implementation of the CPython interpreter? > That > would mean, you always need the whole Jython core/interpreter libs to > create a self contained Jar file. Or, is Jython/Clamp able to create > "native" Java byte code without a Python Interpreter in between? > > Like this: > Python source -> Jython Interpreter -> Jython interprets code and > executes at runtime in a Java VM > Or more like this > Python source -> Java byte code -> Java Interpreter of the VM > interprets > byte code at runtime (Jython not necessary anymore) Jython creates bytecode that's run directly by the jvm, but much of the core pieces of python are implemented in java and are still needed by the generated bytecode. Things like list, dict, int, str and so on are contained in jython's jar, so it's still needed by the compiled modules. So jython isn't interpreting anything, but it still needs some core classes. Charlie |
From: Jan W. <Jan...@et...> - 2009-11-05 07:50:19
|
Hi Charlie, I'm not sure if you're aware of that, but I just described my problem to the jython-users group. And they told me Clamp is exactly what I'm looking for. ;) So, first jython, than clamp, then jython, now clamp again...I'm bit puzzled but I think I understand approximately what Clamp is designed to do. I followed your advise, used compile all, built a JAR file containing the class and the whole Jython lib (I planned to stip it down when it's working) and then passed it to the Java ME emulator. But is sais "can'T find MIDlet class". Then I observed the Java byte code generated by Jython and saw that the resulting class is not implementing MIDlet but PyRunnable and an $1 has been added to the class name. I think, this makes it impossible for the emulator to run the MIDlet. So, I have to admit that I'm not really sure what exactly the "proxy classes" you were talking about are doing. Alex Grönholm told me, that you have not finished to implement all proxy generation hooks. But what are they for? Do they act as correctly named interfaces to other native Java classes? Something like you generate one proxy class for each jython class that looks like a class how it should be in Java but which delivers the calls to subjacent jython classes / methods? If yes, then it really seems, that Clamp should solve my problem, shouldn't it? Whatever, I already offered you my help. So if there is anything I could do, just say it. In your first mail, you explained that your work is half way done but it's still on your working copy and not committed yet. If you like, I'll be glad to have a look on the current code just to get an idea what you are doing and how I could possibly help. You could just send the important classes as a zip file to me. I doesn't have to work, I doesn't need to be compilable and fully commented, it's just to get an overview. Thanks, Jan > -----Ursprüngliche Nachricht----- > Von: Charlie Groves [mailto:cha...@gm...] > Gesendet: Dienstag, 3. November 2009 20:19 > An: Jan Wedel > Cc: jyt...@li... > Betreff: Re: AW: Re: [Jython-dev] Clamp Development > > On Nov 2, 2009, at 3:42, "Jan Wedel" <Jan...@et...> wrote: > > > Hi! > > > >> If you just want to use arbitrary Python modules in Jython without > >> dynamic compilation and don't care about exposing them as Java > >> classes, clamp isn't adding anything. However, one of the things I > >> added to Jython for clamp is the ability to specify where to store > >> the > >> bytecode Jython generates and that stored bytecode should be used > >> instead of always dynamically generating it. That should allow you > >> to > >> use 3rd party libs in a restricted environment. > > > > Do you mean Java byte code or Python byte code? If Python actually > > generates Java byte code, where can I find it? Do you know any > > documentation by chance, that explains how to generate Java class > > files > > or (in an optimal way) create a self-contained jar file from Python > > code? > > It's java byte code. Whenever you import a module with jython, it > spits out a $py.class file corresponding to the imported module in > the same directory. You can use the compileall module included with > jython to compile a whole directory tree and control where it's > output. After compiling everything, you no longer need the .py files. > I think there are a couple guides on the wiki for doing this. > > However jython still generates java bytecode for proxy classes at > runtime, even if the modules containing the proxy classes are > precompiled. This is what the proxy generation branch takes care of. > > >> I'm a little worried about your desire for "native speed". Jython > is > >> a decently fast implementation of Python, but it's not going to be > as > >> fast as plain Java code. I don't have a good feel for what the > ratio > >> is of Java performance to Jython performance these days, especially > >> for VMs other than Sun's, so I can't say how much slower Jython will > >> be. What level of performance do you need? Maybe someone else here > >> can give you an idea of how Jython will do. > > > > I've done some performance tests on my Python Interpreter. It runs > > 200-300 times slower that the same program coded in "native" Java. > > Hmm, > > I think I'm still a bit confused about what Jython does. > > > > Is Jython "only" a Java implementation of the CPython interpreter? > > That > > would mean, you always need the whole Jython core/interpreter libs to > > create a self contained Jar file. Or, is Jython/Clamp able to create > > "native" Java byte code without a Python Interpreter in between? > > > > Like this: > > Python source -> Jython Interpreter -> Jython interprets code and > > executes at runtime in a Java VM > > Or more like this > > Python source -> Java byte code -> Java Interpreter of the VM > > interprets > > byte code at runtime (Jython not necessary anymore) > > Jython creates bytecode that's run directly by the jvm, but much of > the core pieces of python are implemented in java and are still needed > by the generated bytecode. Things like list, dict, int, str and so on > are contained in jython's jar, so it's still needed by the compiled > modules. So jython isn't interpreting anything, but it still needs > some core classes. > > Charlie > |
From: Charlie G. <cha...@gm...> - 2009-11-15 19:25:44
|
Hi Jan, On Wed, Nov 4, 2009 at 11:48 PM, Jan Wedel <Jan...@et...> wrote: > I followed your advise, used compile all, built a JAR file containing > the class and the whole Jython lib (I planned to stip it down when it's > working) and then passed it to the Java ME emulator. But is sais "can'T > find MIDlet class". Then I observed the Java byte code generated by > Jython and saw that the resulting class is not implementing MIDlet but > PyRunnable and an $1 has been added to the class name. I think, this > makes it impossible for the emulator to run the MIDlet. I'm sorry that came off as advice of something to do. What I was trying to say is that jython compiles modules to bytecode which can then be executed without dynamic compilation by PythonInterpreter. They aren't meant to run as regular Java classes as you saw; that's what the proxy classes do. > So, I have to admit that I'm not really sure what exactly the "proxy > classes" you were talking about are doing. Alex Grönholm told me, that > you have not finished to implement all proxy generation hooks. But what > are they for? Do they act as correctly named interfaces to other native > Java classes? Something like you generate one proxy class for each > jython class that looks like a class how it should be in Java but which > delivers the calls to subjacent jython classes / methods? Yes, the proxy classes are the bridges between Java code and Python code. They implement Java interfaces and extend Java classes, and when the methods they override are invoked, they know how to method lookup on a Python instance and invoke the method there. > If yes, then it really seems, that Clamp should solve my problem, > shouldn't it? Actually, clamp isn't needed for your use case. Clamp lets you put decorators on python classes to turn them into something callable from Java. If you're just trying to extend a Java class, you just need the ability to statically compile and name those proxies. Clamp needed that as well, which is why the two keep getting conflated. > Whatever, I already offered you my help. So if there is anything I could > do, just say it. In your first mail, you explained that your work is > half way done but it's still on your working copy and not committed yet. > If you like, I'll be glad to have a look on the current code just to get > an idea what you are doing and how I could possibly help. You could just > send the important classes as a zip file to me. I doesn't have to work, > I doesn't need to be compilable and fully commented, it's just to get an > overview. Well, the majority of the work is already available in the customizable-proxymaker branch in the jython repository. That already contains the static compilation you need. The bit that remains in my local copy is a refactoring of it to allow the generators of proxies to fill in whatever bytecode they like. I think it's a better way to expose all of this, but I've tried a few times over the past weeks to get back to it and haven't been able to scrape together the time. I don't think the work in my local copy is in a state to be comprehensible without a fair amount of explanation, but the stuff committed to the branch compiles and works as far as I know. If you want to pick up from there and get it to a state that it's ready to be merged back to trunk, that would be a way to move this forward. I'd love to get back to this at some point, but it doesn't look like that's going to happen anytime soon, and I don't want to prevent someone else from working on it. The major work remaining there for your use case is a) adding a tool to take advantage of the static proxy compilation and b) using that to visit all the classes in the standard lib that need it and make static proxies for them. Charlie |
From: Jan W. <Jan...@et...> - 2009-11-16 11:11:35
|
> Actually, clamp isn't needed for your use case. Clamp lets you put > decorators on python classes to turn them into something callable from > Java. If you're just trying to extend a Java class, you just need the > ability to statically compile and name those proxies. Clamp needed > that as well, which is why the two keep getting conflated. > > Well, the majority of the work is already available in the > customizable-proxymaker branch in the jython repository. That already > contains the static compilation you need. > The major work remaining there for > your use case is a) adding a tool to take advantage of the static > proxy compilation and b) using that to visit all the classes in the > standard lib that need it and make static proxies for them. Ah, Ok...I have a vague idea of what I could do. But still, despite the fact that it's not completed yet, would't it be easier for me to use clamp by adding annotations to the python code than to write new tools to statically change the class proxies. However, I checked out the code from the branch and I compiled it. But what now? :) I know, to answer such a question would be very time-consuming for you and you already drew a rough sketch. But I am currently sitting in front of huge stack of source files and don't know where to start. I would need some kind of entry point ;) Here are some questions that fly around in my head: 1.) At first, to fulfill a) can I use the resulting jython.jar file from that customizable-proxymaker branch build and create an external app? Or do I have to somehow "hook" into the existing code? If yes, how would I trigger that? By using compileall? What if I don't want to expose all methods? 2.) I think I need to understand the code structure and hierarchy of jython/proxymaker i.e. "what does what and when does it 'that'?" To do that, could you please tell me which class is the main entry point for the proxymaker? I saw that you modified some classes in the org.python.exposed package but couldn't identify a main class from which I could start studying... Thanks in advance! Jan |
From: Jan W. <Jan...@et...> - 2009-11-10 14:26:47
|
Hey Charlie, I guess you're busy right now, but I just want to clarify some things. To make a Jython class file run on a JavaME, they need to be compatible with CLDC and MIDP/IMP. That means the ClassLoader and e.g. JNI is not supported. Moreover, the APIs are very limited (see http://java.sun.com/javame/reference/apis/jsr118/). E.g., HashMap and Templates are not supported. Instead, Hashtable and type casting must be used. That means, there might be some work to do to make it compatible with JavaME. Can you approximate if there is a lot of code needs to be ported? Or even some parts that heavily depend on dynamic class loading (with can not be implemented in JaveME CLDC at all)? This is only important for all Jython/Clamp classes that are needed at runtime on the embedded environment and not interpreter or interactive promt stuff. Maybe you or a Jython dev could comment on that. Thanks, jan > -----Ursprüngliche Nachricht----- > Von: Charlie Groves [mailto:cha...@gm...] > Gesendet: Dienstag, 3. November 2009 20:19 > An: Jan Wedel > Cc: jyt...@li... > Betreff: Re: AW: Re: [Jython-dev] Clamp Development > > On Nov 2, 2009, at 3:42, "Jan Wedel" <Jan...@et...> wrote: > > > Hi! > > > >> If you just want to use arbitrary Python modules in Jython without > >> dynamic compilation and don't care about exposing them as Java > >> classes, clamp isn't adding anything. However, one of the things I > >> added to Jython for clamp is the ability to specify where to store > >> the > >> bytecode Jython generates and that stored bytecode should be used > >> instead of always dynamically generating it. That should allow you > >> to > >> use 3rd party libs in a restricted environment. > > > > Do you mean Java byte code or Python byte code? If Python actually > > generates Java byte code, where can I find it? Do you know any > > documentation by chance, that explains how to generate Java class > > files > > or (in an optimal way) create a self-contained jar file from Python > > code? > > It's java byte code. Whenever you import a module with jython, it > spits out a $py.class file corresponding to the imported module in > the same directory. You can use the compileall module included with > jython to compile a whole directory tree and control where it's > output. After compiling everything, you no longer need the .py files. > I think there are a couple guides on the wiki for doing this. > > However jython still generates java bytecode for proxy classes at > runtime, even if the modules containing the proxy classes are > precompiled. This is what the proxy generation branch takes care of. > > >> I'm a little worried about your desire for "native speed". Jython > is > >> a decently fast implementation of Python, but it's not going to be > as > >> fast as plain Java code. I don't have a good feel for what the > ratio > >> is of Java performance to Jython performance these days, especially > >> for VMs other than Sun's, so I can't say how much slower Jython will > >> be. What level of performance do you need? Maybe someone else here > >> can give you an idea of how Jython will do. > > > > I've done some performance tests on my Python Interpreter. It runs > > 200-300 times slower that the same program coded in "native" Java. > > Hmm, > > I think I'm still a bit confused about what Jython does. > > > > Is Jython "only" a Java implementation of the CPython interpreter? > > That > > would mean, you always need the whole Jython core/interpreter libs to > > create a self contained Jar file. Or, is Jython/Clamp able to create > > "native" Java byte code without a Python Interpreter in between? > > > > Like this: > > Python source -> Jython Interpreter -> Jython interprets code and > > executes at runtime in a Java VM > > Or more like this > > Python source -> Java byte code -> Java Interpreter of the VM > > interprets > > byte code at runtime (Jython not necessary anymore) > > Jython creates bytecode that's run directly by the jvm, but much of > the core pieces of python are implemented in java and are still needed > by the generated bytecode. Things like list, dict, int, str and so on > are contained in jython's jar, so it's still needed by the compiled > modules. So jython isn't interpreting anything, but it still needs > some core classes. > > Charlie > |
From: Charlie G. <cha...@gm...> - 2009-11-15 19:25:49
|
Hi Jan, On Tue, Nov 10, 2009 at 6:25 AM, Jan Wedel <Jan...@et...> wrote: > I guess you're busy right now, but I just want to clarify some things. > To make a Jython class file run on a JavaME, they need to be compatible > with CLDC and MIDP/IMP. That means the ClassLoader and e.g. JNI is not > supported. Moreover, the APIs are very limited (see > http://java.sun.com/javame/reference/apis/jsr118/). E.g., HashMap and > Templates are not supported. Instead, Hashtable and type casting must be > used. That means, there might be some work to do to make it compatible > with JavaME. Can you approximate if there is a lot of code needs to be > ported? Or even some parts that heavily depend on dynamic class loading > (with can not be implemented in JaveME CLDC at all)? This is only > important for all Jython/Clamp classes that are needed at runtime on the > embedded environment and not interpreter or interactive promt stuff. We do use HashMap and generics fairly heavily throughout the codebase. Also, the store for dicts is a ConcurrentHashMap, which while not a HashMap, probably isn't supported either. Since I've never done any programming for JavaME, I'm not sure what else isn't available and what additional effort would be needed. It does sound like there would be a decent amount of work in getting just those basic types working. |