From: Simon T. <st...@ac...> - 2006-07-28 21:04:22
|
Hi all, We're using jython with a fairly large python code base. The initial = jython startup time is pretty slow.=20 The very first startup time (no .class files on disk) is about 10 = seconds. Subsequent startup times are around 7 seconds.=20 So I am guessing the first time is slowest because jython writes .class = files to disk. What is most of the startup time spent in once the .class = files are on disk? Is it jython trying to figure out all the module = dependencies, and whether or not it needs to generate .class files?=20 Is there any way to precompile all modules into .class files using = jythonc, in order to reduce starup time? Thanks. Simon |
From: Kent J. <ke...@td...> - 2006-07-29 11:08:12
|
Simon Toens wrote: > > Hi all, > > We're using jython with a fairly large python code base. The initial > jython startup time is pretty slow. > > The very first startup time (no .class files on disk) is about 10 seconds. > Subsequent startup times are around 7 seconds. > > So I am guessing the first time is slowest because jython writes > .class files to disk. What is most of the startup time spent in once > the .class files are on disk? Is it jython trying to figure out all > the module dependencies, and whether or not it needs to generate > .class files? > I think it is just the imports that are slow. In my larger projects I try to defer imports to shorten the startup time. For example, normal practice is to put all the imports at the top of each module: # module main.py import a def main(): a.do_something() # module a.py import b def do_something(): b.really_do_it() # module b.py def really_do_it(): pass When main.py is run, it imports a.py which imports b.py, so when the program is run it loads every module. This will also extend to importing every Java module used by the program. In contrast, consider this: # module main.py import a def main(): a.do_something() # module a.py def do_something(): import b b.really_do_it() # module b.py def really_do_it(): pass Now b is not imported until it is used. There is no significant performance impact to this, since all imports after the first are essentially dictionary lookups. Kent |
From: Charlie G. <cha...@gm...> - 2006-07-30 00:18:49
|
Do either of you(or anyone out there) have a decent way to reproduce the slow startup time? An application or library that I could import pretty easily? I'd like to start something with a lot of imports under my profiler, but the Jython codebases I have at hand are measly have good startup times. I doubt that I'll actually find anything but it'd be good to check while I'm in the midst of the import code. Thanks, Charlie On 7/29/06, Kent Johnson <ke...@td...> wrote: > Simon Toens wrote: > > > > Hi all, > > > > We're using jython with a fairly large python code base. The initial > > jython startup time is pretty slow. > > > > The very first startup time (no .class files on disk) is about 10 seconds. > > Subsequent startup times are around 7 seconds. > > > > So I am guessing the first time is slowest because jython writes > > .class files to disk. What is most of the startup time spent in once > > the .class files are on disk? Is it jython trying to figure out all > > the module dependencies, and whether or not it needs to generate > > .class files? > > > I think it is just the imports that are slow. In my larger projects I > try to defer imports to shorten the startup time. For example, normal > practice is to put all the imports at the top of each module: > > # module main.py > import a > def main(): > a.do_something() > > # module a.py > import b > def do_something(): > b.really_do_it() > > # module b.py > def really_do_it(): > pass > > When main.py is run, it imports a.py which imports b.py, so when the > program is run it loads every module. This will also extend to importing > every Java module used by the program. > > In contrast, consider this: > > # module main.py > import a > def main(): > a.do_something() > > # module a.py > def do_something(): > import b > b.really_do_it() > > # module b.py > def really_do_it(): > pass > > Now b is not imported until it is used. There is no significant > performance impact to this, since all imports after the first are > essentially dictionary lookups. > > Kent > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev > |
From: Kent J. <ke...@td...> - 2006-07-30 02:42:34
Attachments:
LotsOfImports.py
MakeImports.py
|
Charlie Groves wrote: > Do either of you(or anyone out there) have a decent way to reproduce > the slow startup time? An application or library that I could import > pretty easily? > Unfortunately the my larger Jython projects are proprietary, and I have already taken measures in them to improve the startup times as described in my previous email. I have attached a couple of programs that might be helpful. What I did was comb one of my existing projects for every java import. The first program attached, LotsOfImports.py, just includes all the imports with a simple timing test. It takes about 0.3 sec to run on my computer (Processor x86 Family 6 Model 13 Stepping 8 GenuineIntel ~1995 Mhz). The second program, MakeImports.py, takes the same set of imports and distributes them one to a file, making a chain of files each of which imports something from java, then imports the next file in the chain. The generated program (ImportTest.py) takes a little over 1 sec to run (after the first run, which of course is considerably longer - it took about 11 secs on my machine). These are fairly artificial tests, I hope they are helpful. It's been quite a while since I worried about this stuff. My suspicion was always that importing from jars was the bottleneck, that's why I used all the java imports in the test. You might want to throw in some imports from other jars you have hanging around. I'll admit that 0.3 seconds is not that long in startup, and to add 0.7 sec to import 100 files doesn't seem all that terrible either. But my recollection is that I was able to shave a second or two off the startup time of one project by deferring imports. Kent > I'd like to start something with a lot of imports under my profiler, > but the Jython codebases I have at hand are measly have good startup > times. I doubt that I'll actually find anything but it'd be good to > check while I'm in the midst of the import code. > > Thanks, > Charlie > > On 7/29/06, Kent Johnson <ke...@td...> wrote: > >> Simon Toens wrote: >> >>> Hi all, >>> >>> We're using jython with a fairly large python code base. The initial >>> jython startup time is pretty slow. >>> >>> The very first startup time (no .class files on disk) is about 10 seconds. >>> Subsequent startup times are around 7 seconds. >>> >>> So I am guessing the first time is slowest because jython writes >>> .class files to disk. What is most of the startup time spent in once >>> the .class files are on disk? Is it jython trying to figure out all >>> the module dependencies, and whether or not it needs to generate >>> .class files? >>> >>> >> I think it is just the imports that are slow. In my larger projects I >> try to defer imports to shorten the startup time. For example, normal >> practice is to put all the imports at the top of each module: >> >> # module main.py >> import a >> def main(): >> a.do_something() >> >> # module a.py >> import b >> def do_something(): >> b.really_do_it() >> >> # module b.py >> def really_do_it(): >> pass >> >> When main.py is run, it imports a.py which imports b.py, so when the >> program is run it loads every module. This will also extend to importing >> every Java module used by the program. >> >> In contrast, consider this: >> >> # module main.py >> import a >> def main(): >> a.do_something() >> >> # module a.py >> def do_something(): >> import b >> b.really_do_it() >> >> # module b.py >> def really_do_it(): >> pass >> >> Now b is not imported until it is used. There is no significant >> performance impact to this, since all imports after the first are >> essentially dictionary lookups. >> >> Kent >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Jython-dev mailing list >> Jyt...@li... >> https://lists.sourceforge.net/lists/listinfo/jython-dev >> >> > > > |
From: Charlie G. <cha...@gm...> - 2006-07-30 17:53:03
Attachments:
only_call_getMethods_when_necessary.diff
|
Thanks for the scripts Kent. Unforunately(or fortunately) they didn't turn up anything egregious in the import system. While poking around in the profiler output, I did notice that most of Jython's startup time is spent in PyType calling fillFromClass which spends most of its time calling getMethods on the class representing the type. From what I can tell, filling in the methods on a type using reflection to find what methods are available is the way classic types were built. Newstyle types already have a list of exposed methods so checking all methods is superfluous. Maybe there's a case where exposed methods doesn't contain everything, but at least for newstyle types currently in svn all methods are in exposed_methods. I imagine as long as you're using gexpose.py to make them it'll be safe. The attached patch moves the newstyle vs classic type loading into an if else block in fillFromClass so that getMethods is only called for classic types. This decreases interpreter startup time(running "new PythonInterpreter()" with an uninitialized PySystemState) from 1.3 seconds to .9 seconds on my Mac. I'd be curious to see how this changes on Sun's VM. Charlie On 7/29/06, Kent Johnson <ke...@td...> wrote: > Charlie Groves wrote: > > Do either of you(or anyone out there) have a decent way to reproduce > > the slow startup time? An application or library that I could import > > pretty easily? > > > Unfortunately the my larger Jython projects are proprietary, and I have > already taken measures in them to improve the startup times as described > in my previous email. > > I have attached a couple of programs that might be helpful. What I did > was comb one of my existing projects for every java import. The first > program attached, LotsOfImports.py, just includes all the imports with a > simple timing test. It takes about 0.3 sec to run on my computer > (Processor x86 Family 6 Model 13 Stepping 8 GenuineIntel ~1995 Mhz). > > The second program, MakeImports.py, takes the same set of imports and > distributes them one to a file, making a chain of files each of which > imports something from java, then imports the next file in the chain. > The generated program (ImportTest.py) takes a little over 1 sec to run > (after the first run, which of course is considerably longer - it took > about 11 secs on my machine). > > These are fairly artificial tests, I hope they are helpful. It's been > quite a while since I worried about this stuff. My suspicion was always > that importing from jars was the bottleneck, that's why I used all the > java imports in the test. You might want to throw in some imports from > other jars you have hanging around. I'll admit that 0.3 seconds is not > that long in startup, and to add 0.7 sec to import 100 files doesn't > seem all that terrible either. But my recollection is that I was able to > shave a second or two off the startup time of one project by deferring > imports. > > Kent > > I'd like to start something with a lot of imports under my profiler, > > but the Jython codebases I have at hand are measly have good startup > > times. I doubt that I'll actually find anything but it'd be good to > > check while I'm in the midst of the import code. > > > > Thanks, > > Charlie > > > > On 7/29/06, Kent Johnson <ke...@td...> wrote: > > > >> Simon Toens wrote: > >> > >>> Hi all, > >>> > >>> We're using jython with a fairly large python code base. The initial > >>> jython startup time is pretty slow. > >>> > >>> The very first startup time (no .class files on disk) is about 10 seconds. > >>> Subsequent startup times are around 7 seconds. > >>> > >>> So I am guessing the first time is slowest because jython writes > >>> .class files to disk. What is most of the startup time spent in once > >>> the .class files are on disk? Is it jython trying to figure out all > >>> the module dependencies, and whether or not it needs to generate > >>> .class files? > >>> > >>> > >> I think it is just the imports that are slow. In my larger projects I > >> try to defer imports to shorten the startup time. For example, normal > >> practice is to put all the imports at the top of each module: > >> > >> # module main.py > >> import a > >> def main(): > >> a.do_something() > >> > >> # module a.py > >> import b > >> def do_something(): > >> b.really_do_it() > >> > >> # module b.py > >> def really_do_it(): > >> pass > >> > >> When main.py is run, it imports a.py which imports b.py, so when the > >> program is run it loads every module. This will also extend to importing > >> every Java module used by the program. > >> > >> In contrast, consider this: > >> > >> # module main.py > >> import a > >> def main(): > >> a.do_something() > >> > >> # module a.py > >> def do_something(): > >> import b > >> b.really_do_it() > >> > >> # module b.py > >> def really_do_it(): > >> pass > >> > >> Now b is not imported until it is used. There is no significant > >> performance impact to this, since all imports after the first are > >> essentially dictionary lookups. > >> > >> Kent > >> > >> > >> ------------------------------------------------------------------------- > >> Take Surveys. Earn Cash. Influence the Future of IT > >> Join SourceForge.net's Techsay panel and you'll get the chance to share your > >> opinions on IT & business topics through brief surveys -- and earn cash > >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > >> _______________________________________________ > >> Jython-dev mailing list > >> Jyt...@li... > >> https://lists.sourceforge.net/lists/listinfo/jython-dev > >> > >> > > > > > > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev > > > > |