[htdig] htdig on multiprocessor machines

[Ace <ac...@gm...>]

Hi there!

I'd like to know if it is possible or imaginable to have htdig use more 
than one processor.

I have htdig running on a multiprocessor machine, and whilst the first 
CPU has 99.9% load, the second is idle most of the time.

If there was a way to split the work into some pieces, this might speed 
up the indexing process a lot, I suppose.

The dual PIII-1000 machine I talk about is now indexing for more than 17 
hours CPU-time and not yet finished. The network probably isn't the 
bottleneck. It's connected to the webservers it indexes via Gigabit 
Ethernet.

Well, it's not really a problem if indexing takes 72 hours, but it'd of 
course be nicer if it took less.

So if you have any idea what I could do... let me know.
I'd be willing to test any weird multiprocessor patches on an old 
multiprocessor machine I also have access to. I'd need a version of 
htdig for Solaris 2.6 for that, though (I never managed to compile htdig 
for Solaris). Well, maybe I could also have 2.8 installed, but I'm not sure.

Thanks in advance,

Peter Asemann