Re: [htdig-dev] Re: Performance issue with exclude_urls

[Gilles D. <gr...@sc...>]

According to Joe R. Jah:
> On Wed, 21 Apr 2004, Lachlan Andrew wrote:
> > Date: Wed, 21 Apr 2004 23:13:27 +1000
> > From: Lachlan Andrew <lh...@us...>
> > To: Gilles Detillieux <gr...@sc...>,
>      Christopher Murtagh <chr...@mc...>
> > Cc: htd...@li...
> > Subject: [htdig-dev] Re: Performance issue with exclude_urls
> > 
> > Greetings Gilles + all,
> > 
> > Yes, I agree that we need a more "polished" patch for the 
> > distribution.  I still like my intermediate path:  If *any* server 
> > blocks or URL blocks are used, then the user takes the performance 
> > hit and re-parses each time.  If *no* server/URL blocks are used, we 
> > use Chris's patch.  This should be just as fast as Chris's patch (in 
> > the "3.1-compatibly mode" without server/URL blocks), and just as 
> > flexible as the current status (if blocks are used).  If that can get 
> > ht://Dig fast enough to get into sarge, then I suggest we implement 
> > it first, and then work on Gilles's more complete solution at more 
> > leisure.
> 
> I applied Chris' patch and ran htdig on the same site as before for
> profile; htdig ran ~40% faster than last time;)  Here is the profile:
> 
>  ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/htdig.gmon.exclude_perform.gz
> 
> > A first hack at this (not even compile-tested) is attached, patched 
> > relative to Chris's patched version, so you can see what I mean.  If 
> > people are in favour, I'll try to work on it over the weekend.
> 
> The "slightly-better.0" patch applies, but it does not compile:
> 
> Retriever.cc: In method `int Retriever::IsValidURL(const String &)':
> Retriever.cc:998: `config_server_URL_blocks' undeclared (first use this function)
> Retriever.cc:998: (Each undeclared identifier is reported only once
> Retriever.cc:998: for each function it appears in.)
> gmake[1]: *** [Retriever.o] Error 1

The patch declares config_server_URL_blocks in conf_parser.h, but not
in any header file that Retriever.cc includes.  Try either adding an
include of conf_parser.h to Retriever.cc, or copy the declaration of
config_server_URL_blocks from conf_parser.h to Retriever.cc.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)