Re: [htdig-dev] Re: Residual database errors

[Lachlan A. <lh...@us...>]

Greetings Neal,

On Monday 17 February 2003 06:41, Neal Richter wrote:
> First question:  What were the results with the pagesize @ 32K??

The run I didn't didn't produce any errors, but that doesn't give me=20
much confidence.  The problem is elusive and I haven't yet played=20
around with the data set (using  exclude_urls)  to try to reproduce=20
it, as I have had to with 8k.

>   If you are indexing the same data in the same order every time
> and the target moves with trivial changes of code or flags.. that
> usually means MEMORY CORRUPTION!!!
>   Unfortunately these types of errors are difficult to duplicate on
> other machines/platforms.

True.  But is the order actually the same?  I haven't read the code=20
thoroughly, but I thought  ht://Dig  had a timing mechanism in order=20
not to flood a particular server.  Couldn't that reorder the search=20
depending on how much time is spent on debugging output?

>   Valgrind is a nice open source memory debugger.
>   This will definetly help find a memory error, but will also drown
> you in output sice htdig is pretty bad with memory leaks.. but you
> can dissable leak detection and look only for memory corruption.

Thanks for the tip.  I had used  mpatrol  previously, which seems more=20
powerful, with the corresponding increase in hassle.  Interestingly,=20
it didn't report any leaks or reproduce the error...  I'm rerunning=20
it without  valgrind  now to check that the error is actually=20
repeatable.

>   I'd also like a link to your test data if you can put it up on a
> high bandwidth server.. and a copy of your conf file and
> commandline htdig options. I'd like to run Insure++ on it.

I'll see what I can do.  However, since all the URLs are  file:///s,=20
the actual data set will differ for you, unless you overwrite your=20
current filesystem.  Remember, this is a *very* pernickety bug!

Now that you have suggested some tools, I'll keep playing around until=20
I have something more concrete to report.

Thanks, as always,
Lachlan