Re: [htdig] bottleneck of htdig 3.2.0b4

[Gilles D. <gr...@sc...>]

According to Stefan Nehlsen:
> compared to version 3.1.5 the digging process (htdig) with version 3.2.0b4
> is really slow.
> 
> There are about 32.000 Pages on our servers, 50% are PDFs and some are
> ridiculous big (html files larger 500kB).
> 
> The stable version (3.1.5) needs about 1.5 hours to create the database from
> scratch and the test installation (latest 3.2.0b4) more then 10 hours.
> 
> I know the 3.2 version of htdig is doing things that had been done by htmerge
> in 3.1.5 but it quite slower than running both in the old version.
> 
> htsearch has become much faster in the new version. This is nice but i have
> problems with corrupt databases and so the speed of htdig is important.
> 
> Where ist the bottleneck of htdig (3.2.*) ?

That's a very good question that's been asked before, but never
satisfactorily answered.  The only way to get to the bottom of it would
be to compile the program with profiling enabled, and then see where htdig
is spending most of its time.  The last time anyone did this and reported
back to us, the results were inconclusive because of a strange conflict
between the bundled GNU regex code and the regex code in his C library.

This was on a BSDi system.  If that's the type of system you're running
htdig on, see http://www.htdig.org/FAQ.html#q5.14 for a suggested fix
(this problem doesn't cause core dumps on all BSDI versions, but does
seem to cause odd behaviour nonetheless).

If you're running a different type of system, and are willing to figure
out how to do profiling on it and get some results back to us (along
with information about what type of system it is), we'd appreciate
the feedback.

I think for most others who've given us general feedback on timings,
they've found that 3.2 is about twice as slow as 3.1.5, not the 6-7
times as slow as you've reported.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930