Re: [htdig-dev] Improving speed of htdig on mac os x

[Neal R. <ne...@ri...>]

On Tue, 26 Jul 2005, Geoffrey Hutchison wrote:

> 
> On Jul 26, 2005, at 9:59 AM, Gustave T. Stresen-Reuter wrote:
> 
> > One way to improve the speed of htdig on mac os x is to take  
> > advantage of the AltiVec processor. I don't know anything about how  
> > to do this, but in reading some documentation I ran across a  
> > document says that "Existing C code written for serial execution"  
> > can take advantage of the AltiVec processor.
> 
> I'm not sure there's really a lot that can happen here. Neal can  
> contradict me, but the last time I did benchmarking/profiling on the  
> code, most of the slowdown during indexing was in the database code,  
> and a large part of that was I/O bound.

  What is your wordlist_cache_size set to?  Make sure that it is about 
2-3% of the expected index size.

  I would also disable all index compression and run htdig again to see if 
you notice a speed up.  Setting wordlist_compress & wordlist_compress_zlib 
to 'false' is the way to test that.
 
> Furthermore, AltiVec really shines doing floating-point processing,  
> e.g. matrix multiplication. There's very, very little of that in ht:// 
> Dig.
> 
> I think for future development (i.e., 4.0) multi-threading the  
> indexing would probably help *MUCH* more than adding AltiVec or SSE  
> processor-specific optimizations which mostly help with floating- 
> point ops.
> 
> And of course, moving to a different database backend would probably  
> help immensely too.

  Yep.

-- 
Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485