Re: [htdig-dev] profiling

[didier <dga...@ma...>]

Hi everybody

Greg Lepore wrote:
> 
>         I am willing to enable profiling on my recent install of the 12/9 build of
> 3.2 beta 4 to figure out some bottlenecks.  I just need to know how and
> what to look for in the results.  I have 10 or so collections ranging from
> 200 - 500,000 documents.  I tried to search the site but the search has
> been down for a couple of days (Internal Server Error).   The system specs
> are: RedHat 7.2,  dual 1GHz processors, 1GB of RAM, 5 18GB SCSI hard
> drives, four are RAID 5 (with /opt/ on that partition).  If you guys can
> tell me how to re-compile and what to look for I can do it.  My initial
> digs (over an NFS mount) were taking from 8-10 times as long as 3.1.3.
>         My rough re-design of the site is still available at:
> http://rhobard.com/htdig/
> I still like it....
> 
I'm afraid profiling won't help a lot.
First look at CPU usage with top, vmstat, time, whatever. If you have a
cpu load around 10, 15% then the bottleneck is IO subsystem not the CPU.
I didn't use 3.2 for a long time (I had 1, 2% CPU load), but if db size
> RAM then you're toasted.

Anyway for profiling (don't know if it's the right way but should work).
make distclean
export CFLAGS=-pg
export CXXFLAGS=-pg
./configure --enable-shared=false
and so on

Don't use rundig (you won't get profile info for htdig) rather use 
htdig -i  and other options
after a long time. 
gprof htdig (you should have a gmon.out in your current dir).


Geoff what about for -i:
going the old way, ascii --> sort --> htmerge.
or
htdig uses a tempory db (half RAM size?) and bulk insert in the
background, if there's such a thing with berkeley db.


Didier