[htdig] Re: HTDIG-3.1.6 / Slow htsearch

[Gilles D. <gr...@sc...>]

Hi, Patrick.  Please have a look at http://www.htdig.org/FAQ.html#q1.16

According to Huby, Patrick:
>     We have recently installed HTDIG-3.1.6 (security pb) and manually
> patched HTDIG program to include your suggestion.
>                            http://www.htdig.org/mail/2000/10/0215.html
> 
>     We have noticed that HTSEARCH is much slower than in previous 3.1.5
> version.

There's very little that's changed in htsearch from 3.1.5 to 3.1.6 that
would cause it to slow down at all.  I certainly can't reproduce any
such problem here.

>     Description of our pb:
> 
>       When we search (using HTSEARCH through a command line) a word for the
> first time , HTSEARCH is very slow. We we seach the same word for the second
> time HTSEARCH is much quicker. The same search with 3.1.5 gives a normal
> response time. 

What you describe is pretty much the standard behaviour you'd expect from
disk caching.  I tried a search four times on my system, only I tried
3.1.5 first, twice, and then 3.1.6 twice, and the first search with
3.1.5 took much longer than the other 3 (as much as 15 times as long).
Even after waiting almost half an hour and trying again, after activities
that caused a lot more disk accesses, the search was almost 3 times as
fast as the first one.  Why don't you try your tests again, after lots
of disk activity has cleared the cache, only try 3.1.5 first, and see if
it isn't slower than the subsequent tests with 3.1.6.

>     As the patch above is not connected to HTSEARCH ,we feel that a number
> of modifications have been made in HTSEARCH (apart the security fix ).
>    Could you confirm this ?

Yes, the patch above only affects htdig, not htsearch.  There have been
a number of changes to htsearch since Feb 2000 (see the ChangeLog in the
3.1.6 snapshot), but most are pretty small and "low impact" as far as
performance are concerned.  I can't imagine how any of these changes could
account for a big hit on search speed, and I haven't observed such a problem.

>    We  would also like to know a way to debug HTSEARCH (debug options on a
> command line gives no error information). 

What sort of information would you like htsearch to put out?  We tend to
add debugging information as needed, to help with specific debugging issues,
so if htsearch's debugging output is pretty sparce it's because it hasn't
required the heavy sort of debugging that htdig has.

>    We are using a big database:
> 
> -rw-rw-r--    1 root     root     857611264 Oct 22 09:57 db.docdb.work
> -rw-rw-r--    1 root     root     14241792 Oct 22 09:57 db.docs.index.work
> -rw-rw-r--    1 root     root     970989189 Oct 22 10:02 db.wordlist.work
> -rw-rw-r--    1 root     root     867992576 Oct 22 10:05 db.words.db.work

You do realise that htsearch doesn't use .work files, don't you?
I assume you moved these into place before running htsearch on them.
Big databases certainly do slow htsearch down, but then that was the
case with older versions as well.  See http://www.htdig.org/FAQ.html#q5.10

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930