Re: [htdig-dev] Re: Residual database problems

[Neal R. <ne...@ri...>]

On Tue, 4 Feb 2003, Lachlan Andrew wrote:

> On Saturday 01 February 2003 11:25, Neal Richter wrote:
>
> > If your error is repeatable, can you test it with
> > wordlist_compress_zlib & wordlist_compress dissabled and re-run
> > htpurge?  I'd like to see if the error still appears.
>
> The diagnostics only appear if compression is enabled.
>
> It takes over 24 hours (which becoems two days...) for the whole dig,
> and I'll try to find a smaller data set that causes the problem.  In
> the mean time, are there any other tests I can do to try to track it
> down?  For example, is it possible to decompress the file offline to
> compare it against the uncompressed one?

  Other than using htdump on both the zlib compressed WordDB and the
uncompressed WordDB I can't think of one.

  If there are differences in the dumps we could then see what effect it
has on searching.  If there are no differences I would lean towards a bug in
htpurge.

You may be able to hack htdump later to only uncompress the
BDB pages in question.

  Have you stepped through the htpurge code to see when/how this happens?

  I am skeptical that it's caused by the zlib compression code because:

1)  The changes to mp_cmpr are very minor to enable zlib compression and
involve no changes to the input data that would otherwise goto the mifluz
page compressor.

2)  If the compressed data was getting corrupted zlib should
report an error on the uncompress of that data-page.

http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/htdig/htdig/db/mp_cmpr.c.diff?r1=1.2&r2=1.3

Thanks.. I appreciate your effort on this.

One you get the htdumps tested for differences please report back!

Neal Richter
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485