Re: [Gramps-devel] slow gramps import

[Peter L. <pet...@te...>]

Finally some else has run into my old "slow problem"!

I did some test:
I have two versions of Gramps on my Windows box:
1. Gramps 3.2
    GRAMPS: 3.2.1-0.SVN15091M 
    Python: 2.6.4 (r264:75708, Oct 26 2009, 08:23:19... 
    BSDDB3: 5.0.0 
    LANG: sv_SE.utf-8
    OS: win32
Here an inmport of a 1.7 MB .gramps file takes about 3 minutes.

2. Gramps 3.3  
    GRAMPS: 3.3.0-0.SVN15837M 
    Python: 2.6.4 (r264:75708, Oct 26 2009, 08:23:19... 
    BSDDB: 4.7.3 (4, 7, 25) 
    LANG: sv_SE.utf-8
    OS: win32
Here an inmport of a 1.7 MB .gramps file takes "forever".
    Reached 4% after 3 minutes.
    Reached 25% after 15 minutes.

I managed to have bsddb3 running in windows but not on Linux.
I ran into problem in Linux installing bsddb3.


3. On My Linux box:
    GRAMPS: 3.2.4-0.SVN15861M 
    Python: 2.6.5 (r265:79063, Jul 14 2010, 13:26:04... 
    BSDDB: 4.7.3 (4, 7, 25) 
    LANG: sv_SE.UTF-8
    OS: Linux
    Distribution: 2.6.33.7-desktop-1mnb
    
it is also slow. It took 16 minutes, but it's a much faster processor.

4. On my second Linux sysrem:
    GRAMPS: 3.2.4-0.SVN15856M 
    Python: 2.6.5 (r265:79063, Jul 14 2010, 13:26:04... 
    BSDDB: 4.7.3 (4, 7, 25) 
    LANG: sv_SE.UTF-8
    OS: Linux
    Distribution: 2.6.33.7-desktop-1mnb
        
The same import took about 5 minutes. This processor is slower than 3. above.
Strange, faster import on a slower system?

/Peter

> ok, forget previous mail, the slowdown must be in the bsddb C core, as put
> does not use iteration (we do use that elsewhere though).
> Hopefully it is fixed in eg ubuntu 10.10
> 
> Benny
> 
> 2010/9/11 Benny Malengier <ben...@gm...>
> 
> > 2010/9/11 Benny Malengier <ben...@gm...>
> > 
> > Hi,
> > 
> >> I notice import of gramps only uses 6 to 8% of my cpu, and is quite slow
> >> in branch32. I do not remember it being so slow before. Anybody know of
> >> changes that could be to blame?
> >> 
> >> Benny
> > 
> > Ok, it seems the windows slow import problem has arrived in linux too, at
> > least ubuntu 10.04.
> > 
> >    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
> >    
> >     15823  156.362    0.010  157.233    0.010 dbshelve.py:247(put)
> > 
> > There is a bug in bsddb dbshelve since 4.7.2, and ubuntu uses 4.7.3.
> > I think from the release notes that this is fixed in 4.8.4, see
> > http://www.jcea.es/programacion/pybsddb.htm:
> > Old regression: dbshelve objects are iterable again. The bug was
> > introduced in pybsddb 4.7.2. Added relevant testcases.
> > 
> > Is there anybody with 4.8.4 to verify it is much faster? Peter, did you
> > now install that manually?
> > Unfortunately, it seems ubuntu 10.10 will use 4.8.3:
> > http://packages.ubuntu.com/maverick/python-bsddb3
> > Is anybody testing Maverick to see it is indeed still just as slow as
> > 10.04?
> > Doug, can you ask ubuntu to upgrade the package to 4.8.4?
> > 
> > To know what pybsddb you are running:
> > benny@dell:~$ python
> > Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
> > [GCC 4.4.3] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > 
> > >>> import bsddb
> > >>> bsddb.__version__
> > 
> > '4.7.3'
> > 
> > 
> > Benny

-- 
Peter Landgren
Talken Hagen	
671 94  BRUNSKOG
0570-530 21
070-345 0964
pet...@te...
Skype: pgl4820.2