From: Gilles D. <gr...@sc...> - 2001-12-11 22:25:12
|
According to Stefan Nehlsen: > compared to version 3.1.5 the digging process (htdig) with version 3.2.0b4 > is really slow. > > There are about 32.000 Pages on our servers, 50% are PDFs and some are > ridiculous big (html files larger 500kB). > > The stable version (3.1.5) needs about 1.5 hours to create the database from > scratch and the test installation (latest 3.2.0b4) more then 10 hours. > > I know the 3.2 version of htdig is doing things that had been done by htmerge > in 3.1.5 but it quite slower than running both in the old version. > > htsearch has become much faster in the new version. This is nice but i have > problems with corrupt databases and so the speed of htdig is important. > > Where ist the bottleneck of htdig (3.2.*) ? That's a very good question that's been asked before, but never satisfactorily answered. The only way to get to the bottom of it would be to compile the program with profiling enabled, and then see where htdig is spending most of its time. The last time anyone did this and reported back to us, the results were inconclusive because of a strange conflict between the bundled GNU regex code and the regex code in his C library. This was on a BSDi system. If that's the type of system you're running htdig on, see http://www.htdig.org/FAQ.html#q5.14 for a suggested fix (this problem doesn't cause core dumps on all BSDI versions, but does seem to cause odd behaviour nonetheless). If you're running a different type of system, and are willing to figure out how to do profiling on it and get some results back to us (along with information about what type of system it is), we'd appreciate the feedback. I think for most others who've given us general feedback on timings, they've found that 3.2 is about twice as slow as 3.1.5, not the 6-7 times as slow as you've reported. -- Gilles R. Detillieux E-mail: <gr...@sc...> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil Dept. Physiology, U. of Manitoba Phone: (204)789-3766 Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930 |