Re: [htdig] htdig on multiprocessor machines

[Geoff H. <ghu...@ws...>]

At 2:57 AM +0100 1/14/02, Ace wrote:
>I have htdig running on a multiprocessor machine, and whilst the 
>first CPU has 99.9% load, the second is idle most of the time.
>
>If there was a way to split the work into some pieces, this might 
>speed up the indexing process a lot, I suppose.

Indexing is certainly not multithreaded, so there's no "SMP-enabled" 
htdig. Besides, the speed of your indexing is largely dependent on 
the speed of the network connection and the speed of the disk serving 
the database in many cases.

If you can come up with ways of splitting the URLs into "chunks," you 
can consider running two htdig indexing process together. Then run 
htmerge on each database and then use the htmerge -m to merge one 
into the other (or each one into another database).

<http://www.htdig.org/FAQ.html#q4.5>

-- 
--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/