[htdig-dev] Residual database problems

[Lachlan A. <lh...@us...>]

Greetings Neal,

Isn't  wordlist_compress_zlib  turned on by default?  I've had a few=20
minor problems recently without changing it.  The one I've got log=20
files for is that  htpurge  displays about six lots of diagnostics of=20
the form

pg->type:  0
************************************
************************************
************************************
page size:8192
 00-07: Log sequence number.  file  : 0
 00-07: Log sequence number.  offset: 0
 08-11: Current page number.  : 143319
 12-15: Previous page number. : 0
 16-19: Next page number.     : 132296
 20-21: Number of item pairs on the page. : 0
 22-23: High free byte page offset.       : 8192
    24: Btree tree level.                 : 0
    25: Page type.                        : 0
entry offsets:
    0:  0  0  0  0  0  0  0  0 d7 2f  2  0  0  0  0  0 c8  4  2  0=20
   20:  0  0  0 20  0  0 fc 1f fc 1f ec 1f e8 1f d8 1f d4 1f c4 1f=20
   40: c0 1f b0 1f ac 1f 9c 1f 98 1f 88 1f 84 1f 74 1f 70 1f 60 1f=20
   60: 5c 1f 4c 1f 48 1f 38 1f 34 1f 24 1f 20 1f 10 1f  c 1f fc 1e=20
   80: f8 1e e8 1e e4 1e d4 1e d0 1e c0 1e bc 1e ac 1e a8 1e 98 1e=20
  100: 94 1e 84 1e 80 1e 70 1e 6c 1e 5c 1e 58 1e 48 1e 44 1e 34 1e
=2E..
=2E..
=2E..

while it is discarding words.  I assumed this is caused by a=20
recoverable error. The only difference in the entries are the=20
current/next pages, and bytes 8, 9, 16, 17, 18 of the "entry=20
offsets".  The first "next page number" is 0, and subsequent ones are=20
the previous values of "current page number" (as if it is reading=20
backwards through a chain).  If you like, I can send the whole 6MB=20
log file, and/or any configuration files you want.

A while ago (but I *think* with your fix in place), I also had a=20
problem with  htdig  crashing at one point, but I've lost the=20
details.  When I've had problems, it has normally been on 10+ hour=20
digs.  Any tips for isolating them?  I've been thinking of doing an=20
integrity check every 100 database writes or so, but I don't know the=20
code well enough yet...

Thanks for your feedback, and very much for writng the _zlib fix!

Cheers,
Lachlan

On Tuesday 28 January 2003 09:38, Neal Richter wrote:
> =09What DB errors are you speaking of?  Turning on
> wordlist_compress_zlib should be a workaround for the DB errors I
> know about.

> > Am I correct in believing that the hold-up is basically
> > database errors?