From: Jim P. <jim...@gm...> - 2013-06-03 22:26:04
|
I'm currently working on an open source Android build tool called Buck which is implemented in Java and uses Python to define build rules (https://github.com/facebook/buck) We recently started embedding Jython in Buck to avoid relying on a system Python and have seen performance gains for big projects which have to parse many build files, but the Jython startup time and memory overhead has made Buck slower for small projects. We're currently using the JSR-223 API to embed Jython and were wondering whether we might see performance gains by switching to using the PythonInterpreter API and or PyObjects (as described here http://www.jython.org/jythonbook/en/1.0/JythonAndJavaIntegration.html#using-jython-within-java-applications) (The current embedding code can be found here https://github.com/facebook/buck/blob/master/src/com/facebook/buck/json/BuildFileToJsonParser.java and the Python which loads build files is here https://github.com/facebook/buck/blob/master/src/com/facebook/buck/parser/buck.py) Thanks! Jim |
From: Emmanuel J. <emm...@gm...> - 2013-06-04 08:28:44
|
Hi, If I may share my experience on this (running jython 2.5.1/jdk 7).As for you, startup time is key for me. Doing some dummy measurement most of the time spent (as expected) seems to be the startup of the JVM. So I guess using you own interpreter won't change perf that much. I do not know who is the culprit but another point I've noticed (doing quick comparison between start time of the jython launcher and my app ) is that startup time is also impacted by the number of jars I add to the classpath. So what I've done on my side is to shrink number of jars to the minimum and and use as much as possible lazy loading of module and Java classes. Again, I've only did quick dummy measurement so let's way for actual jython dev guy to give you proper answer to this. hope this helps. EJ On Tue, Jun 4, 2013 at 12:25 AM, Jim Purbrick <jim...@gm...> wrote: > I'm currently working on an open source Android build tool called Buck > which is implemented in Java and uses Python to define build rules > (https://github.com/facebook/buck) > > We recently started embedding Jython in Buck to avoid relying on a > system Python and have seen performance gains for big projects which > have to parse many build files, but the Jython startup time and memory > overhead has made Buck slower for small projects. > > We're currently using the JSR-223 API to embed Jython and were > wondering whether we might see performance gains by switching to using > the PythonInterpreter API and or PyObjects (as described here > > http://www.jython.org/jythonbook/en/1.0/JythonAndJavaIntegration.html#using-jython-within-java-applications > ) > > (The current embedding code can be found here > > https://github.com/facebook/buck/blob/master/src/com/facebook/buck/json/BuildFileToJsonParser.java > and the Python which loads build files is here > > https://github.com/facebook/buck/blob/master/src/com/facebook/buck/parser/buck.py > ) > > Thanks! > > Jim > > > ------------------------------------------------------------------------------ > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer high-level views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenow-d2d-j > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > |
From: Walter C. <cha...@gm...> - 2013-06-04 12:22:54
|
Most of jython's startup overhead comes from: -jvm startup Did you try starting the jvm in client mode? -initial caching of jars in your classpath bootpath pruning your classpath does help a bit on initial runs. -try the openjdk 8 jigsaw preview releases this will allow you to shrink your jvm somewhat. Note that openjdk hotspot is slower than the oracle version. -jvm startup memory options a big heap slows the startup of the jre. Small builds probably can start with less memory HTH, -W On Jun 4, 2013 4:32 AM, "Emmanuel Jannetti" <emm...@gm...> wrote: > Hi, > > If I may share my experience on this (running jython 2.5.1/jdk 7).As for > you, startup time is key for me. Doing some dummy measurement > most of the time spent (as expected) seems to be the startup of the JVM. > So I guess using you own interpreter won't > change perf that much. > I do not know who is the culprit but another point I've noticed (doing > quick comparison between start time of the jython launcher and my app ) is > that startup time is also impacted by the number of jars I add to the > classpath. > So what I've done on my side is to shrink number of jars to the minimum > and and use as much as possible lazy loading of module and Java classes. > > Again, I've only did quick dummy measurement so let's way for actual > jython dev guy to give you proper answer to this. > > hope this helps. > EJ > > > > > On Tue, Jun 4, 2013 at 12:25 AM, Jim Purbrick <jim...@gm...>wrote: > >> I'm currently working on an open source Android build tool called Buck >> which is implemented in Java and uses Python to define build rules >> (https://github.com/facebook/buck) >> >> We recently started embedding Jython in Buck to avoid relying on a >> system Python and have seen performance gains for big projects which >> have to parse many build files, but the Jython startup time and memory >> overhead has made Buck slower for small projects. >> >> We're currently using the JSR-223 API to embed Jython and were >> wondering whether we might see performance gains by switching to using >> the PythonInterpreter API and or PyObjects (as described here >> >> http://www.jython.org/jythonbook/en/1.0/JythonAndJavaIntegration.html#using-jython-within-java-applications >> ) >> >> (The current embedding code can be found here >> >> https://github.com/facebook/buck/blob/master/src/com/facebook/buck/json/BuildFileToJsonParser.java >> and the Python which loads build files is here >> >> https://github.com/facebook/buck/blob/master/src/com/facebook/buck/parser/buck.py >> ) >> >> Thanks! >> >> Jim >> >> >> ------------------------------------------------------------------------------ >> How ServiceNow helps IT people transform IT departments: >> 1. A cloud service to automate IT design, transition and operations >> 2. Dashboards that offer high-level views of enterprise services >> 3. A single system of record for all IT processes >> http://p.sf.net/sfu/servicenow-d2d-j >> _______________________________________________ >> Jython-users mailing list >> Jyt...@li... >> https://lists.sourceforge.net/lists/listinfo/jython-users >> > > > > ------------------------------------------------------------------------------ > How ServiceNow helps IT people transform IT departments: > 1. A cloud service to automate IT design, transition and operations > 2. Dashboards that offer high-level views of enterprise services > 3. A single system of record for all IT processes > http://p.sf.net/sfu/servicenow-d2d-j > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > > |
From: Jim P. <jim...@gm...> - 2013-06-17 15:28:51
|
On 4 June 2013 14:22, Walter Chang <cha...@gm...> wrote: > Most of jython's startup overhead comes from: > -jvm startup > Did you try starting the jvm in client mode? Yes: my test command ran slightly slower (32s vs 30s in server mode) > -initial caching of jars in your classpath bootpath > pruning your classpath does help a bit on initial runs. How do I do this and still have java find the jars when it needs them? Currently I have 28 jars passed to java via "-classpath". > -try the openjdk 8 jigsaw preview releases > this will allow you to shrink your jvm somewhat. Note that openjdk hotspot > is slower than the oracle version. I'd like to avoid jdk specific optimizations if possible. > -jvm startup memory options > a big heap slows the startup of the jre. Small builds probably can start > with less memory I tried passing -Xmx600m, -Xmx300m and -Xmx128m to java and saw nearly identical run times compared with the default -Xmx1000m (but significantly lower memory usage) Are there any other options I can try? Would using the PythonInterpreter API and/or PyObjects APIs instead of JSR-223 help? Could I prune Python libraries that aren't used out of Jython to improve performance? Cheers, Jim |
From: Jim B. <jb...@zy...> - 2013-06-17 16:39:50
|
TL;DR - use the bootpath, this is the easiest win On Mon, Jun 17, 2013 at 9:28 AM, Jim Purbrick <jim...@gm...> wrote: > On 4 June 2013 14:22, Walter Chang <cha...@gm...> wrote: > > Most of jython's startup overhead comes from: > > -jvm startup > > Did you try starting the jvm in client mode? > > Yes: my test command ran slightly slower (32s vs 30s in server mode) > > > -initial caching of jars in your classpath bootpath > > pruning your classpath does help a bit on initial runs. > > How do I do this and still have java find the jars when it needs them? > Currently I have 28 jars passed to java via "-classpath". > Use -Xbootclasspath/a as an alternative to -classpath for those jars (FWIW, I'd recommend looking at the jython script for this and other similar environmental setup). This is probably the easiest way to get a significant win on Jython startup performance by avoiding the overhead of class verification. So $ time jython27 -c "print 42" 42 real 0m1.363s user 0m1.976s sys 0m0.150s vs $ time jython27 --boot -c "print 42" 42 real 0m0.919s user 0m1.212s sys 0m0.115s (jython27 is my alias to the jython script running against Jython trunk; lastly all numbers are reported from my laptop... differences are certainly significant and stable) > > -try the openjdk 8 jigsaw preview releases > > this will allow you to shrink your jvm somewhat. Note that openjdk > hotspot > > is slower than the oracle version. > > I'd like to avoid jdk specific optimizations if possible. > > > -jvm startup memory options > > a big heap slows the startup of the jre. Small builds probably can start > > with less memory > > I tried passing -Xmx600m, -Xmx300m and -Xmx128m to java and saw nearly > identical run times compared with the default -Xmx1000m (but > significantly lower memory usage) > > Are there any other options I can try? Would using the > PythonInterpreter API and/or PyObjects APIs instead of JSR-223 help? > JSR223 support and PythonInterpreter implement similar facades to the Jython runtime, therefore I would not expect even observable performance differences. > Could I prune Python libraries that aren't used out of Jython to > improve performance? > The closest to this is -S (don't import site, and those dependencies) in the jython startup script, however this path is not used by JSR223/PythonInterpreter. Savings of ~250 ms in my testing: $ time jython27 --boot -S -c "print 42" 42 real 0m0.673s user 0m0.861s sys 0m0.093s - Jim |
From: Walter C. <cha...@gm...> - 2013-06-18 02:09:18
|
Some projects have embedded nailgun with success as well http://www.martiansoftware.com/nailgun/ What version of the jvm are you running? When running a 64-bit-only Hotspot JVM, there is no client mode. You can get the same effect as client mode by passing two flags: -XX:+TieredCompilation -XX:TieredStopAtLevel=1 Jim's suggestions with bootpath and -S are also good. I've experimented with -Xverify:none with varying success as well in the past. But nailgun seems to give the most bang for your efforts if that integration is something you can consider. -W On Jun 17, 2013 12:39 PM, "Jim Baker" <jb...@zy...> wrote: > TL;DR - use the bootpath, this is the easiest win > > On Mon, Jun 17, 2013 at 9:28 AM, Jim Purbrick <jim...@gm...>wrote: > >> On 4 June 2013 14:22, Walter Chang <cha...@gm...> wrote: >> > Most of jython's startup overhead comes from: >> > -jvm startup >> > Did you try starting the jvm in client mode? >> >> Yes: my test command ran slightly slower (32s vs 30s in server mode) >> >> > -initial caching of jars in your classpath bootpath >> > pruning your classpath does help a bit on initial runs. >> >> How do I do this and still have java find the jars when it needs them? >> Currently I have 28 jars passed to java via "-classpath". >> > > Use -Xbootclasspath/a as an alternative to -classpath for those jars > (FWIW, I'd recommend looking at the jython script for this and other > similar environmental setup). This is probably the easiest way to get a > significant win on Jython startup performance by avoiding the overhead of > class verification. > > So > > $ time jython27 -c "print 42" > 42 > > real 0m1.363s > user 0m1.976s > sys 0m0.150s > > vs > > $ time jython27 --boot -c "print 42" > 42 > > real 0m0.919s > user 0m1.212s > sys 0m0.115s > > (jython27 is my alias to the jython script running against Jython trunk; > lastly all numbers are reported from my laptop... differences are certainly > significant and stable) > > >> > -try the openjdk 8 jigsaw preview releases >> > this will allow you to shrink your jvm somewhat. Note that openjdk >> hotspot >> > is slower than the oracle version. >> >> I'd like to avoid jdk specific optimizations if possible. >> >> > -jvm startup memory options >> > a big heap slows the startup of the jre. Small builds probably can >> start >> > with less memory >> >> I tried passing -Xmx600m, -Xmx300m and -Xmx128m to java and saw nearly >> identical run times compared with the default -Xmx1000m (but >> significantly lower memory usage) >> >> Are there any other options I can try? Would using the >> PythonInterpreter API and/or PyObjects APIs instead of JSR-223 help? >> > > JSR223 support and PythonInterpreter implement similar facades to the > Jython runtime, therefore I would not expect even observable performance > differences. > > >> Could I prune Python libraries that aren't used out of Jython to >> improve performance? >> > > The closest to this is -S (don't import site, and those dependencies) in > the jython startup script, however this path is not used by > JSR223/PythonInterpreter. Savings of ~250 ms in my testing: > > $ time jython27 --boot -S -c "print 42" > 42 > > real 0m0.673s > user 0m0.861s > sys 0m0.093s > > - Jim > |
From: Walter C. <cha...@gm...> - 2013-06-18 03:23:56
|
http://wiki.python.org/jython/PackageScanning Your comments about 28 jars means that the pkg manager will cache the jars on the first startup which takes quite a bit of time to process for the first run. If you only use full class imports, you can skip the package scanning altogether. Set the system property python.cachedir.skip to true and it should start much more quickly than it currently does (unless you are using jythonstandalone.jar which skips caching already) -W On Jun 17, 2013 10:09 PM, "Walter Chang" <cha...@gm...> wrote: > Some projects have embedded nailgun with success as well > > http://www.martiansoftware.com/nailgun/ > > What version of the jvm are you running? > When running a 64-bit-only Hotspot JVM, there is no client mode. You can > get the same effect as client mode by passing two flags: > -XX:+TieredCompilation -XX:TieredStopAtLevel=1 > > Jim's suggestions with bootpath and -S are also good. > > I've experimented with -Xverify:none with varying success as well in the > past. > > But nailgun seems to give the most bang for your efforts if that > integration is something you can consider. > > -W > On Jun 17, 2013 12:39 PM, "Jim Baker" <jb...@zy...> wrote: > >> TL;DR - use the bootpath, this is the easiest win >> >> On Mon, Jun 17, 2013 at 9:28 AM, Jim Purbrick <jim...@gm...>wrote: >> >>> On 4 June 2013 14:22, Walter Chang <cha...@gm...> wrote: >>> > Most of jython's startup overhead comes from: >>> > -jvm startup >>> > Did you try starting the jvm in client mode? >>> >>> Yes: my test command ran slightly slower (32s vs 30s in server mode) >>> >>> > -initial caching of jars in your classpath bootpath >>> > pruning your classpath does help a bit on initial runs. >>> >>> How do I do this and still have java find the jars when it needs them? >>> Currently I have 28 jars passed to java via "-classpath". >>> >> >> Use -Xbootclasspath/a as an alternative to -classpath for those jars >> (FWIW, I'd recommend looking at the jython script for this and other >> similar environmental setup). This is probably the easiest way to get a >> significant win on Jython startup performance by avoiding the overhead of >> class verification. >> >> So >> >> $ time jython27 -c "print 42" >> 42 >> >> real 0m1.363s >> user 0m1.976s >> sys 0m0.150s >> >> vs >> >> $ time jython27 --boot -c "print 42" >> 42 >> >> real 0m0.919s >> user 0m1.212s >> sys 0m0.115s >> >> (jython27 is my alias to the jython script running against Jython trunk; >> lastly all numbers are reported from my laptop... differences are certainly >> significant and stable) >> >> >>> > -try the openjdk 8 jigsaw preview releases >>> > this will allow you to shrink your jvm somewhat. Note that openjdk >>> hotspot >>> > is slower than the oracle version. >>> >>> I'd like to avoid jdk specific optimizations if possible. >>> >>> > -jvm startup memory options >>> > a big heap slows the startup of the jre. Small builds probably can >>> start >>> > with less memory >>> >>> I tried passing -Xmx600m, -Xmx300m and -Xmx128m to java and saw nearly >>> identical run times compared with the default -Xmx1000m (but >>> significantly lower memory usage) >>> >>> Are there any other options I can try? Would using the >>> PythonInterpreter API and/or PyObjects APIs instead of JSR-223 help? >>> >> >> JSR223 support and PythonInterpreter implement similar facades to the >> Jython runtime, therefore I would not expect even observable performance >> differences. >> >> >>> Could I prune Python libraries that aren't used out of Jython to >>> improve performance? >>> >> >> The closest to this is -S (don't import site, and those dependencies) in >> the jython startup script, however this path is not used by >> JSR223/PythonInterpreter. Savings of ~250 ms in my testing: >> >> $ time jython27 --boot -S -c "print 42" >> 42 >> >> real 0m0.673s >> user 0m0.861s >> sys 0m0.093s >> >> - Jim >> > |