From: Peter L. <pet...@te...> - 2010-03-16 12:01:46
|
The import speed was discussed a little in http://www.gramps-project.org/bugs/view.php?id=3609 Now, I have downgraded to Python 2.5.4 with the recommended in http://www.gramps-project.org/wiki/index.php?title=Windows_installer and now the XML import is as fast as on Linux. /Peter |
From: Gerald B. <ger...@gm...> - 2010-03-16 12:51:01
|
Wow. Is it just Python, or the bsddb modules or something else? On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: > The import speed was discussed a little in > http://www.gramps-project.org/bugs/view.php?id=3609 > > Now, I have downgraded to Python 2.5.4 with the recommended in > http://www.gramps-project.org/wiki/index.php?title=Windows_installer > > and now the XML import is as fast as on Linux. > > /Peter > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Gramps-devel mailing list > Gra...@li... > https://lists.sourceforge.net/lists/listinfo/gramps-devel > -- Gerald Britton |
From: Peter L. <pet...@te...> - 2010-03-16 12:55:43
|
There is no speed problem when I do a check and repair. /Peter > Wow. Is it just Python, or the bsddb modules or something else? > > On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: > > The import speed was discussed a little in > > http://www.gramps-project.org/bugs/view.php?id=3609 > > > > Now, I have downgraded to Python 2.5.4 with the recommended in > > http://www.gramps-project.org/wiki/index.php?title=Windows_installer > > > > and now the XML import is as fast as on Linux. > > > > /Peter > > > > > > ------------------------------------------------------------------------- > >----- Download Intel® Parallel Studio Eval > > Try the new software tools for yourself. Speed compiling, find bugs > > proactively, and fine-tune applications for parallel performance. > > See why Intel Parallel Studio got high marks during beta. > > http://p.sf.net/sfu/intel-sw-dev > > _______________________________________________ > > Gramps-devel mailing list > > Gra...@li... > > https://lists.sourceforge.net/lists/listinfo/gramps-devel |
From: Gerald B. <ger...@gm...> - 2010-03-16 12:58:18
|
Nice to know, but I don't see how it pertains to import. Are you saying that check and repair runs about the same on Python 2.6 and 2.5? On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: > There is no speed problem when I do a check and repair. > /Peter > >> Wow. Is it just Python, or the bsddb modules or something else? >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: >> > The import speed was discussed a little in >> > http://www.gramps-project.org/bugs/view.php?id=3609 >> > >> > Now, I have downgraded to Python 2.5.4 with the recommended in >> > http://www.gramps-project.org/wiki/index.php?title=Windows_installer >> > >> > and now the XML import is as fast as on Linux. >> > >> > /Peter >> > >> > >> > ------------------------------------------------------------------------- >> >----- Download Intel® Parallel Studio Eval >> > Try the new software tools for yourself. Speed compiling, find bugs >> > proactively, and fine-tune applications for parallel performance. >> > See why Intel Parallel Studio got high marks during beta. >> > http://p.sf.net/sfu/intel-sw-dev >> > _______________________________________________ >> > Gramps-devel mailing list >> > Gra...@li... >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel > > -- Gerald Britton |
From: Peter L. <pet...@te...> - 2010-03-16 13:02:09
|
Just a subjective feeling doing it. But I could do a timing experiment later today. /Peter > Nice to know, but I don't see how it pertains to import. Are you > saying that check and repair runs about the same on Python 2.6 and > 2.5? > > On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: > > There is no speed problem when I do a check and repair. > > /Peter > > > >> Wow. Is it just Python, or the bsddb modules or something else? > >> > >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: > >> > The import speed was discussed a little in > >> > http://www.gramps-project.org/bugs/view.php?id=3609 > >> > > >> > Now, I have downgraded to Python 2.5.4 with the recommended in > >> > http://www.gramps-project.org/wiki/index.php?title=Windows_installer > >> > > >> > and now the XML import is as fast as on Linux. > >> > > >> > /Peter > >> > > >> > > >> > ---------------------------------------------------------------------- > >> >--- ----- Download Intel® Parallel Studio Eval > >> > Try the new software tools for yourself. Speed compiling, find bugs > >> > proactively, and fine-tune applications for parallel performance. > >> > See why Intel Parallel Studio got high marks during beta. > >> > http://p.sf.net/sfu/intel-sw-dev > >> > _______________________________________________ > >> > Gramps-devel mailing list > >> > Gra...@li... > >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel -- Peter Landgren Talken Hagen 671 94 BRUNSKOG 0570-530 21 070-345 0964 pet...@te... Skype: pgl4820.2 |
From: Gerald B. <ger...@gm...> - 2010-03-16 13:13:16
|
Probably don't need timings. It's the order of magnitude that I'm worried about. If check-and-repair runs about the same, I'd say we can rule out problems with bsddb and Python itself. Let's see: What else does import use? from xml.parsers.expat Could this be the culprit? I can't find anything talking about performance problems with it, though. On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: > Just a subjective feeling doing it. But I could do a timing experiment later today. > /Peter > > >> Nice to know, but I don't see how it pertains to import. Are you >> saying that check and repair runs about the same on Python 2.6 and >> 2.5? >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: >> > There is no speed problem when I do a check and repair. >> > /Peter >> > >> >> Wow. Is it just Python, or the bsddb modules or something else? >> >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: >> >> > The import speed was discussed a little in >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 >> >> > >> >> > Now, I have downgraded to Python 2.5.4 with the recommended in >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows_installer >> >> > >> >> > and now the XML import is as fast as on Linux. >> >> > >> >> > /Peter >> >> > >> >> > >> >> > ---------------------------------------------------------------------- >> >> >--- ----- Download Intel® Parallel Studio Eval >> >> > Try the new software tools for yourself. Speed compiling, find bugs >> >> > proactively, and fine-tune applications for parallel performance. >> >> > See why Intel Parallel Studio got high marks during beta. >> >> > http://p.sf.net/sfu/intel-sw-dev >> >> > _______________________________________________ >> >> > Gramps-devel mailing list >> >> > Gra...@li... >> >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel > > -- > Peter Landgren > Talken Hagen > 671 94 BRUNSKOG > 0570-530 21 > 070-345 0964 > pet...@te... > Skype: pgl4820.2 > > -- Gerald Britton |
From: Peter L. <pet...@te...> - 2010-03-16 13:26:22
|
OK. I ran; - check and repair and it took about 1 min. - reconstruct sec index took about 15 sec - rebuild ref maps took > 20 minutes! rebuild took 20 seconds on my Linux box with the same database. /Peter > Probably don't need timings. It's the order of magnitude that I'm > worried about. If check-and-repair runs about the same, I'd say we > can rule out problems with bsddb and Python itself. Let's see: What > else does import use? > > from xml.parsers.expat > > Could this be the culprit? I can't find anything talking about > performance problems with it, though. > > On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: > > Just a subjective feeling doing it. But I could do a timing experiment > > later today. /Peter > > > >> Nice to know, but I don't see how it pertains to import. Are you > >> saying that check and repair runs about the same on Python 2.6 and > >> 2.5? > >> > >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: > >> > There is no speed problem when I do a check and repair. > >> > /Peter > >> > > >> >> Wow. Is it just Python, or the bsddb modules or something else? > >> >> > >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: > >> >> > The import speed was discussed a little in > >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 > >> >> > > >> >> > Now, I have downgraded to Python 2.5.4 with the recommended in > >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows_installe > >> >> >r > >> >> > > >> >> > and now the XML import is as fast as on Linux. > >> >> > > >> >> > /Peter > >> >> > > >> >> > > >> >> > ------------------------------------------------------------------- > >> >> >--- --- ----- Download Intel® Parallel Studio Eval > >> >> > Try the new software tools for yourself. Speed compiling, find bugs > >> >> > proactively, and fine-tune applications for parallel performance. > >> >> > See why Intel Parallel Studio got high marks during beta. > >> >> > http://p.sf.net/sfu/intel-sw-dev > >> >> > _______________________________________________ > >> >> > Gramps-devel mailing list > >> >> > Gra...@li... > >> >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel > > > > -- > > Peter Landgren > > Talken Hagen > > 671 94 BRUNSKOG > > 0570-530 21 > > 070-345 0964 > > pet...@te... > > Skype: pgl4820.2 -- Peter Landgren Talken Hagen 671 94 BRUNSKOG 0570-530 21 070-345 0964 pet...@te... Skype: pgl4820.2 |
From: Gerald B. <ger...@gm...> - 2010-03-16 13:37:43
|
totally weird. I can't see how it can be our code, which is pretty much OS-agnostic in these areas. I'm wondering if the Windows library ports use memory properly. To rebuild the ref maps means reading everything in the db to see what is related to what. If most of the data is in memory, this would be very fast. If it has to go to disk again and again ... well, you can see the implications I'm sure. On Tue, Mar 16, 2010 at 9:26 AM, Peter Landgren <pet...@te...> wrote: > OK. > I ran; > - check and repair and it took about 1 min. > - reconstruct sec index took about 15 sec > - rebuild ref maps took > 20 minutes! > rebuild took 20 seconds on my Linux box with the same database. > /Peter > > >> Probably don't need timings. It's the order of magnitude that I'm >> worried about. If check-and-repair runs about the same, I'd say we >> can rule out problems with bsddb and Python itself. Let's see: What >> else does import use? >> >> from xml.parsers.expat >> >> Could this be the culprit? I can't find anything talking about >> performance problems with it, though. >> >> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: >> > Just a subjective feeling doing it. But I could do a timing experiment >> > later today. /Peter >> > >> >> Nice to know, but I don't see how it pertains to import. Are you >> >> saying that check and repair runs about the same on Python 2.6 and >> >> 2.5? >> >> >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: >> >> > There is no speed problem when I do a check and repair. >> >> > /Peter >> >> > >> >> >> Wow. Is it just Python, or the bsddb modules or something else? >> >> >> >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: >> >> >> > The import speed was discussed a little in >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 >> >> >> > >> >> >> > Now, I have downgraded to Python 2.5.4 with the recommended in >> >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows_installe >> >> >> >r >> >> >> > >> >> >> > and now the XML import is as fast as on Linux. >> >> >> > >> >> >> > /Peter >> >> >> > >> >> >> > >> >> >> > ------------------------------------------------------------------- >> >> >> >--- --- ----- Download Intel® Parallel Studio Eval >> >> >> > Try the new software tools for yourself. Speed compiling, find bugs >> >> >> > proactively, and fine-tune applications for parallel performance. >> >> >> > See why Intel Parallel Studio got high marks during beta. >> >> >> > http://p.sf.net/sfu/intel-sw-dev >> >> >> > _______________________________________________ >> >> >> > Gramps-devel mailing list >> >> >> > Gra...@li... >> >> >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel >> > >> > -- >> > Peter Landgren >> > Talken Hagen >> > 671 94 BRUNSKOG >> > 0570-530 21 >> > 070-345 0964 >> > pet...@te... >> > Skype: pgl4820.2 > > -- > Peter Landgren > Talken Hagen > 671 94 BRUNSKOG > 0570-530 21 > 070-345 0964 > pet...@te... > Skype: pgl4820.2 > > -- Gerald Britton |
From: Peter L. <pet...@te...> - 2010-03-16 13:43:02
|
Indeed. The windows box is a Windows 7 64 bits 3 GB of memory. If I look at the progress bar down to the left, it's goes fast up to 66% and stays there for almost the rest of the reconstruction. As I said with the XML import the disk LED light was ON all the time, indicating intensive disk activity. /Peter > totally weird. I can't see how it can be our code, which is pretty > much OS-agnostic in these areas. > > I'm wondering if the Windows library ports use memory properly. To > rebuild the ref maps means reading everything in the db to see what is > related to what. If most of the data is in memory, this would be very > fast. If it has to go to disk again and again ... well, you can see > the implications I'm sure. > > On Tue, Mar 16, 2010 at 9:26 AM, Peter Landgren <pet...@te...> wrote: > > OK. > > I ran; > > - check and repair and it took about 1 min. > > - reconstruct sec index took about 15 sec > > - rebuild ref maps took > 20 minutes! > > rebuild took 20 seconds on my Linux box with the same database. > > /Peter > > > >> Probably don't need timings. It's the order of magnitude that I'm > >> worried about. If check-and-repair runs about the same, I'd say we > >> can rule out problems with bsddb and Python itself. Let's see: What > >> else does import use? > >> > >> from xml.parsers.expat > >> > >> Could this be the culprit? I can't find anything talking about > >> performance problems with it, though. > >> > >> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: > >> > Just a subjective feeling doing it. But I could do a timing experiment > >> > later today. /Peter > >> > > >> >> Nice to know, but I don't see how it pertains to import. Are you > >> >> saying that check and repair runs about the same on Python 2.6 and > >> >> 2.5? > >> >> > >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: > >> >> > There is no speed problem when I do a check and repair. > >> >> > /Peter > >> >> > > >> >> >> Wow. Is it just Python, or the bsddb modules or something else? > >> >> >> > >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: > >> >> >> > The import speed was discussed a little in > >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 > >> >> >> > > >> >> >> > Now, I have downgraded to Python 2.5.4 with the recommended in > >> >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows_insta > >> >> >> >lle r > >> >> >> > > >> >> >> > and now the XML import is as fast as on Linux. > >> >> >> > > >> >> >> > /Peter |
From: Peter L. <pet...@te...> - 2010-03-16 18:30:30
|
Is there any parameter in the bsddb, that I could do some experiments with? > totally weird. I can't see how it can be our code, which is pretty > much OS-agnostic in these areas. > > I'm wondering if the Windows library ports use memory properly. To > rebuild the ref maps means reading everything in the db to see what is > related to what. If most of the data is in memory, this would be very > fast. If it has to go to disk again and again ... well, you can see > the implications I'm sure. > > On Tue, Mar 16, 2010 at 9:26 AM, Peter Landgren <pet...@te...> wrote: > > OK. > > I ran; > > - check and repair and it took about 1 min. > > - reconstruct sec index took about 15 sec > > - rebuild ref maps took > 20 minutes! > > rebuild took 20 seconds on my Linux box with the same database. > > /Peter > > > >> Probably don't need timings. It's the order of magnitude that I'm > >> worried about. If check-and-repair runs about the same, I'd say we > >> can rule out problems with bsddb and Python itself. Let's see: What > >> else does import use? > >> > >> from xml.parsers.expat > >> > >> Could this be the culprit? I can't find anything talking about > >> performance problems with it, though. > >> > >> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: > >> > Just a subjective feeling doing it. But I could do a timing experiment > >> > later today. /Peter > >> > > >> >> Nice to know, but I don't see how it pertains to import. Are you > >> >> saying that check and repair runs about the same on Python 2.6 and > >> >> 2.5? > >> >> > >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: > >> >> > There is no speed problem when I do a check and repair. > >> >> > /Peter > >> >> > > >> >> >> Wow. Is it just Python, or the bsddb modules or something else? > >> >> >> > >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: > >> >> >> > The import speed was discussed a little in > >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 > >> >> >> > > >> >> >> > Now, I have downgraded to Python 2.5.4 with the recommended in > >> >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows_insta > >> >> >> >lle r > >> >> >> > > >> >> >> > and now the XML import is as fast as on Linux. > >> >> >> > > >> >> >> > /Peter |
From: Gerald B. <ger...@gm...> - 2010-03-16 18:46:21
|
Look in src/gen/db/dbconst.py. There are a few constants there. The most obvious might be the cache size. On Tue, Mar 16, 2010 at 2:30 PM, Peter Landgren <pet...@te...> wrote: > Is there any parameter in the bsddb, that I could do some experiments with? > >> totally weird. I can't see how it can be our code, which is pretty >> much OS-agnostic in these areas. >> >> I'm wondering if the Windows library ports use memory properly. To >> rebuild the ref maps means reading everything in the db to see what is >> related to what. If most of the data is in memory, this would be very >> fast. If it has to go to disk again and again ... well, you can see >> the implications I'm sure. >> >> On Tue, Mar 16, 2010 at 9:26 AM, Peter Landgren <pet...@te...> wrote: >> > OK. >> > I ran; >> > - check and repair and it took about 1 min. >> > - reconstruct sec index took about 15 sec >> > - rebuild ref maps took > 20 minutes! >> > rebuild took 20 seconds on my Linux box with the same database. >> > /Peter >> > >> >> Probably don't need timings. It's the order of magnitude that I'm >> >> worried about. If check-and-repair runs about the same, I'd say we >> >> can rule out problems with bsddb and Python itself. Let's see: What >> >> else does import use? >> >> >> >> from xml.parsers.expat >> >> >> >> Could this be the culprit? I can't find anything talking about >> >> performance problems with it, though. >> >> >> >> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: >> >> > Just a subjective feeling doing it. But I could do a timing experiment >> >> > later today. /Peter >> >> > >> >> >> Nice to know, but I don't see how it pertains to import. Are you >> >> >> saying that check and repair runs about the same on Python 2.6 and >> >> >> 2.5? >> >> >> >> >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: >> >> >> > There is no speed problem when I do a check and repair. >> >> >> > /Peter >> >> >> > >> >> >> >> Wow. Is it just Python, or the bsddb modules or something else? >> >> >> >> >> >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: >> >> >> >> > The import speed was discussed a little in >> >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 >> >> >> >> > >> >> >> >> > Now, I have downgraded to Python 2.5.4 with the recommended in >> >> >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows_insta >> >> >> >> >lle r >> >> >> >> > >> >> >> >> > and now the XML import is as fast as on Linux. >> >> >> >> > >> >> >> >> > /Peter > > -- Gerald Britton |
From: Jérôme <rom...@ya...> - 2010-03-17 15:47:02
|
Serge also gave some tips : > "One solution is to create a DB_CONFIG file in the gramps database directory. > for this, you create a new database. > you stop gramps. > In the new database directory, you create the DB_CONFIG file. > In this file, you put the following lines : > > set_lk_max_locks 50000 > set_lk_max_objects 50000 > > Then you restart gramps and import your big database. " http://www.gramps-project.org/bugs/view.php?id=2686#c8362 > "I remember another thing. > bsddb use shared memory for index. So we can go over the limits. > If we are in this following case : > modify or add the following lines in /etc/sysctl.conf : > > kernel.maxshm = 2147483647 > or > kernel/maxshm = 2147483647 > > These two lines are identicaly. This means you can put a dot or a slash between kernel and shmmax. > 2147483647 means 2GB. you can set a lower value depending of you RAM and swap. > > Once this is done, either reboot or as root launch the command : > sysctl -p > > This value will be reloaded at next boot." http://www.gramps-project.org/bugs/view.php?id=2686#c8365 Note, I got a crash some weeks ago, because there was a lot of log files after a DB upgrade (import XML, generated under Gramps 3.0.x). Maybe no more free space (or cache ?). Gerald Britton a écrit : > Look in src/gen/db/dbconst.py. There are a few constants there. The > most obvious might be the cache size. > > On Tue, Mar 16, 2010 at 2:30 PM, Peter Landgren <pet...@te...> wrote: >> Is there any parameter in the bsddb, that I could do some experiments with? >> >>> totally weird. I can't see how it can be our code, which is pretty >>> much OS-agnostic in these areas. >>> >>> I'm wondering if the Windows library ports use memory properly. To >>> rebuild the ref maps means reading everything in the db to see what is >>> related to what. If most of the data is in memory, this would be very >>> fast. If it has to go to disk again and again ... well, you can see >>> the implications I'm sure. >>> >>> On Tue, Mar 16, 2010 at 9:26 AM, Peter Landgren <pet...@te...> wrote: >>>> OK. >>>> I ran; >>>> - check and repair and it took about 1 min. >>>> - reconstruct sec index took about 15 sec >>>> - rebuild ref maps took > 20 minutes! >>>> rebuild took 20 seconds on my Linux box with the same database. >>>> /Peter >>>> >>>>> Probably don't need timings. It's the order of magnitude that I'm >>>>> worried about. If check-and-repair runs about the same, I'd say we >>>>> can rule out problems with bsddb and Python itself. Let's see: What >>>>> else does import use? >>>>> >>>>> from xml.parsers.expat >>>>> >>>>> Could this be the culprit? I can't find anything talking about >>>>> performance problems with it, though. >>>>> >>>>> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: >>>>>> Just a subjective feeling doing it. But I could do a timing experiment >>>>>> later today. /Peter >>>>>> >>>>>>> Nice to know, but I don't see how it pertains to import. Are you >>>>>>> saying that check and repair runs about the same on Python 2.6 and >>>>>>> 2.5? >>>>>>> >>>>>>> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: >>>>>>>> There is no speed problem when I do a check and repair. >>>>>>>> /Peter >>>>>>>> >>>>>>>>> Wow. Is it just Python, or the bsddb modules or something else? >>>>>>>>> >>>>>>>>> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: >>>>>>>>>> The import speed was discussed a little in >>>>>>>>>> http://www.gramps-project.org/bugs/view.php?id=3609 >>>>>>>>>> >>>>>>>>>> Now, I have downgraded to Python 2.5.4 with the recommended in >>>>>>>>>> http://www.gramps-project.org/wiki/index.php?title=Windows_insta >>>>>>>>>> lle r >>>>>>>>>> >>>>>>>>>> and now the XML import is as fast as on Linux. >>>>>>>>>> >>>>>>>>>> /Peter >> > > > |
From: Peter L. <pet...@te...> - 2010-03-17 13:35:12
|
I have tested with the cache 16 times larger and 16 times smaller and no significant change in speed. As the Gramps code is the same, the speed problem must be in Python 2.6 or the bsddb library. /Peter > Look in src/gen/db/dbconst.py. There are a few constants there. The > most obvious might be the cache size. > > On Tue, Mar 16, 2010 at 2:30 PM, Peter Landgren <pet...@te...> wrote: > > Is there any parameter in the bsddb, that I could do some experiments > > with? > > > >> totally weird. I can't see how it can be our code, which is pretty > >> much OS-agnostic in these areas. > >> > >> I'm wondering if the Windows library ports use memory properly. To > >> rebuild the ref maps means reading everything in the db to see what is > >> related to what. If most of the data is in memory, this would be very > >> fast. If it has to go to disk again and again ... well, you can see > >> the implications I'm sure. > >> > >> > OK. > >> > I ran; > >> > - check and repair and it took about 1 min. > >> > - reconstruct sec index took about 15 sec > >> > - rebuild ref maps took > 20 minutes! > >> > rebuild took 20 seconds on my Linux box with the same database. > >> > /Peter > >> > > >> >> Probably don't need timings. It's the order of magnitude that I'm > >> >> worried about. If check-and-repair runs about the same, I'd say we > >> >> can rule out problems with bsddb and Python itself. Let's see: What > >> >> else does import use? > >> >> > >> >> from xml.parsers.expat > >> >> > >> >> Could this be the culprit? I can't find anything talking about > >> >> performance problems with it, though. > >> >> > >> >> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: > >> >> > Just a subjective feeling doing it. But I could do a timing > >> >> > experiment later today. /Peter > >> >> > > >> >> >> Nice to know, but I don't see how it pertains to import. Are you > >> >> >> saying that check and repair runs about the same on Python 2.6 and > >> >> >> 2.5? > >> >> >> > >> >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: > >> >> >> > There is no speed problem when I do a check and repair. > >> >> >> > /Peter > >> >> >> > > >> >> >> >> Wow. Is it just Python, or the bsddb modules or something > >> >> >> >> else? > >> >> >> >> > >> >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: > >> >> >> >> > The import speed was discussed a little in > >> >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 > >> >> >> >> > > >> >> >> >> > Now, I have downgraded to Python 2.5.4 with the recommended > >> >> >> >> > in > >> >> >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows_in > >> >> >> >> >sta lle r > >> >> >> >> > > >> >> >> >> > and now the XML import is as fast as on Linux. > >> >> >> >> > > >> >> >> >> > /Peter |
From: Benny M. <ben...@gm...> - 2010-03-17 15:12:41
|
2010/3/17 Peter Landgren <pet...@te...>: > I have tested with the cache 16 times larger and 16 times smaller and no significant change in speed. > As the Gramps code is the same, the speed problem must be in Python 2.6 or the bsddb library. or the compilation flags. Perhaps they forgot the -O flag :-) There is little more you can do than indicating the problem. It takes an expert with valgrind or other tools to find the reason of the problem. Benny > > /Peter > >> Look in src/gen/db/dbconst.py. There are a few constants there. The >> most obvious might be the cache size. >> >> On Tue, Mar 16, 2010 at 2:30 PM, Peter Landgren <pet...@te...> wrote: >> > Is there any parameter in the bsddb, that I could do some experiments >> > with? >> > >> >> totally weird. I can't see how it can be our code, which is pretty >> >> much OS-agnostic in these areas. >> >> >> >> I'm wondering if the Windows library ports use memory properly. To >> >> rebuild the ref maps means reading everything in the db to see what is >> >> related to what. If most of the data is in memory, this would be very >> >> fast. If it has to go to disk again and again ... well, you can see >> >> the implications I'm sure. >> >> >> >> > OK. >> >> > I ran; >> >> > - check and repair and it took about 1 min. >> >> > - reconstruct sec index took about 15 sec >> >> > - rebuild ref maps took > 20 minutes! >> >> > rebuild took 20 seconds on my Linux box with the same database. >> >> > /Peter >> >> > >> >> >> Probably don't need timings. It's the order of magnitude that I'm >> >> >> worried about. If check-and-repair runs about the same, I'd say we >> >> >> can rule out problems with bsddb and Python itself. Let's see: What >> >> >> else does import use? >> >> >> >> >> >> from xml.parsers.expat >> >> >> >> >> >> Could this be the culprit? I can't find anything talking about >> >> >> performance problems with it, though. >> >> >> >> >> >> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: >> >> >> > Just a subjective feeling doing it. But I could do a timing >> >> >> > experiment later today. /Peter >> >> >> > >> >> >> >> Nice to know, but I don't see how it pertains to import. Are you >> >> >> >> saying that check and repair runs about the same on Python 2.6 and >> >> >> >> 2.5? >> >> >> >> >> >> >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: >> >> >> >> > There is no speed problem when I do a check and repair. >> >> >> >> > /Peter >> >> >> >> > >> >> >> >> >> Wow. Is it just Python, or the bsddb modules or something >> >> >> >> >> else? >> >> >> >> >> >> >> >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: >> >> >> >> >> > The import speed was discussed a little in >> >> >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 >> >> >> >> >> > >> >> >> >> >> > Now, I have downgraded to Python 2.5.4 with the recommended >> >> >> >> >> > in >> >> >> >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows_in >> >> >> >> >> >sta lle r >> >> >> >> >> > >> >> >> >> >> > and now the XML import is as fast as on Linux. >> >> >> >> >> > >> >> >> >> >> > /Peter > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Gramps-devel mailing list > Gra...@li... > https://lists.sourceforge.net/lists/listinfo/gramps-devel > |
From: Peter L. <pet...@te...> - 2010-03-17 16:39:26
|
Maybe it's beyond the scope of Gramps to try to debug this. BTW, is valgrind available for Windows 7 64 bits, which would be needed in this case. /Peter > 2010/3/17 Peter Landgren <pet...@te...>: > > I have tested with the cache 16 times larger and 16 times smaller and no > > significant change in speed. As the Gramps code is the same, the speed > > problem must be in Python 2.6 or the bsddb library. > > or the compilation flags. Perhaps they forgot the -O flag :-) > > There is little more you can do than indicating the problem. It takes > an expert with valgrind or other tools to find the reason of the > problem. > > Benny > > > /Peter > > > >> Look in src/gen/db/dbconst.py. There are a few constants there. The > >> most obvious might be the cache size. > >> > >> On Tue, Mar 16, 2010 at 2:30 PM, Peter Landgren <pet...@te...> wrote: > >> > Is there any parameter in the bsddb, that I could do some experiments > >> > with? > >> > > >> >> totally weird. I can't see how it can be our code, which is pretty > >> >> much OS-agnostic in these areas. > >> >> > >> >> I'm wondering if the Windows library ports use memory properly. To > >> >> rebuild the ref maps means reading everything in the db to see what > >> >> is related to what. If most of the data is in memory, this would be > >> >> very fast. If it has to go to disk again and again ... well, you can > >> >> see the implications I'm sure. > >> >> > >> >> > OK. > >> >> > I ran; > >> >> > - check and repair and it took about 1 min. > >> >> > - reconstruct sec index took about 15 sec > >> >> > - rebuild ref maps took > 20 minutes! > >> >> > rebuild took 20 seconds on my Linux box with the same database. > >> >> > /Peter > >> >> > > >> >> >> Probably don't need timings. It's the order of magnitude that I'm > >> >> >> worried about. If check-and-repair runs about the same, I'd say > >> >> >> we can rule out problems with bsddb and Python itself. Let's see: > >> >> >> What else does import use? > >> >> >> > >> >> >> from xml.parsers.expat > >> >> >> > >> >> >> Could this be the culprit? I can't find anything talking about > >> >> >> performance problems with it, though. > >> >> >> > >> >> >> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: > >> >> >> > Just a subjective feeling doing it. But I could do a timing > >> >> >> > experiment later today. /Peter > >> >> >> > > >> >> >> >> Nice to know, but I don't see how it pertains to import. Are > >> >> >> >> you saying that check and repair runs about the same on Python > >> >> >> >> 2.6 and 2.5? > >> >> >> >> > >> >> >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: > >> >> >> >> > There is no speed problem when I do a check and repair. > >> >> >> >> > /Peter > >> >> >> >> > > >> >> >> >> >> Wow. Is it just Python, or the bsddb modules or something > >> >> >> >> >> else? > >> >> >> >> >> > >> >> >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: > >> >> >> >> >> > The import speed was discussed a little in > >> >> >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 > >> >> >> >> >> > > >> >> >> >> >> > Now, I have downgraded to Python 2.5.4 with the > >> >> >> >> >> > recommended in > >> >> >> >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows > >> >> >> >> >> >_in sta lle r > >> >> >> >> >> > > >> >> >> >> >> > and now the XML import is as fast as on Linux. > >> >> >> >> >> > > >> >> >> >> >> > /Peter |
From: Gerald B. <ger...@gm...> - 2010-03-17 16:41:06
|
I'd say beyond the scope of Gramps On Wed, Mar 17, 2010 at 12:39 PM, Peter Landgren <pet...@te...> wrote: > Maybe it's beyond the scope of Gramps to try to debug this. > BTW, is valgrind available for Windows 7 64 bits, which would be needed in this case. > > /Peter > >> 2010/3/17 Peter Landgren <pet...@te...>: >> > I have tested with the cache 16 times larger and 16 times smaller and no >> > significant change in speed. As the Gramps code is the same, the speed >> > problem must be in Python 2.6 or the bsddb library. >> >> or the compilation flags. Perhaps they forgot the -O flag :-) >> >> There is little more you can do than indicating the problem. It takes >> an expert with valgrind or other tools to find the reason of the >> problem. >> >> Benny >> >> > /Peter >> > >> >> Look in src/gen/db/dbconst.py. There are a few constants there. The >> >> most obvious might be the cache size. >> >> >> >> On Tue, Mar 16, 2010 at 2:30 PM, Peter Landgren <pet...@te...> wrote: >> >> > Is there any parameter in the bsddb, that I could do some experiments >> >> > with? >> >> > >> >> >> totally weird. I can't see how it can be our code, which is pretty >> >> >> much OS-agnostic in these areas. >> >> >> >> >> >> I'm wondering if the Windows library ports use memory properly. To >> >> >> rebuild the ref maps means reading everything in the db to see what >> >> >> is related to what. If most of the data is in memory, this would be >> >> >> very fast. If it has to go to disk again and again ... well, you can >> >> >> see the implications I'm sure. >> >> >> >> >> >> > OK. >> >> >> > I ran; >> >> >> > - check and repair and it took about 1 min. >> >> >> > - reconstruct sec index took about 15 sec >> >> >> > - rebuild ref maps took > 20 minutes! >> >> >> > rebuild took 20 seconds on my Linux box with the same database. >> >> >> > /Peter >> >> >> > >> >> >> >> Probably don't need timings. It's the order of magnitude that I'm >> >> >> >> worried about. If check-and-repair runs about the same, I'd say >> >> >> >> we can rule out problems with bsddb and Python itself. Let's see: >> >> >> >> What else does import use? >> >> >> >> >> >> >> >> from xml.parsers.expat >> >> >> >> >> >> >> >> Could this be the culprit? I can't find anything talking about >> >> >> >> performance problems with it, though. >> >> >> >> >> >> >> >> On Tue, Mar 16, 2010 at 9:01 AM, Peter Landgren <pet...@te...> wrote: >> >> >> >> > Just a subjective feeling doing it. But I could do a timing >> >> >> >> > experiment later today. /Peter >> >> >> >> > >> >> >> >> >> Nice to know, but I don't see how it pertains to import. Are >> >> >> >> >> you saying that check and repair runs about the same on Python >> >> >> >> >> 2.6 and 2.5? >> >> >> >> >> >> >> >> >> >> On Tue, Mar 16, 2010 at 8:55 AM, Peter Landgren <pet...@te...> wrote: >> >> >> >> >> > There is no speed problem when I do a check and repair. >> >> >> >> >> > /Peter >> >> >> >> >> > >> >> >> >> >> >> Wow. Is it just Python, or the bsddb modules or something >> >> >> >> >> >> else? >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Mar 16, 2010 at 8:01 AM, Peter Landgren <pet...@te...> wrote: >> >> >> >> >> >> > The import speed was discussed a little in >> >> >> >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3609 >> >> >> >> >> >> > >> >> >> >> >> >> > Now, I have downgraded to Python 2.5.4 with the >> >> >> >> >> >> > recommended in >> >> >> >> >> >> > http://www.gramps-project.org/wiki/index.php?title=Windows >> >> >> >> >> >> >_in sta lle r >> >> >> >> >> >> > >> >> >> >> >> >> > and now the XML import is as fast as on Linux. >> >> >> >> >> >> > >> >> >> >> >> >> > /Peter > > -- Gerald Britton |