[CLucene-dev] Strange Assertion Error. clucene 2.3.3.4

[Teryl T. <ter...@gm...>]

Hi there,

I've been playing around with clucene (great piece of software by the way)
and I seem to be getting a random assertional error every once in a while
when I'm writing out indexes.    The assertional error is as follows:

Assertion failed: (doc < numDocsInRAM), function appendPostings, file
/Users/terylt/Projects/libs/clucene/src/core/CLucene/index/DocumentsWriter.cpp,
line 759.

The stack trace of the error is as follows:

#12 0x00018daa in ConnectionBuffer::AddConnection (this=0x3b7f180,
conn=0x337d880) at
/Users/terylt/Projects/Canaris2/trunk/mega_collector/MegaCollector/ConnectionBuffer.cpp:410
(gdb) frame 10
#10 0x0022c518 in lucene::index::IndexWriter::addDocument (this=0x3b7f310,
doc=0x0, analyzer=0x0) at
/Users/terylt/Projects/Canaris2/libs/clucene/src/core/CLucene/index/IndexWriter.cpp:702
(gdb) frame 9
#9  0x00229ed3 in lucene::index::IndexWriter::flush (this=0x3b7f310,
triggerMerge=true, _flushDocStores=false) at
/Users/terylt/Projects/Canaris2/libs/clucene/src/core/CLucene/index/IndexWriter.cpp:1330
(gdb) frame 8
#8  lucene::index::IndexWriter::doFlush (this=0x3b7f310,
_flushDocStores=false) at
/Users/terylt/Projects/Canaris2/libs/clucene/src/core/CLucene/index/IndexWriter.cpp:0
(gdb) frame 7
#7  std::string::c_str () at basic_string.h:1559
(gdb) frame 6
#6  0x001eb24f in lucene::index::DocumentsWriter::flush (this=0x45f5b430,
_closeDocStore=true) at
/Users/terylt/Projects/Canaris2/libs/clucene/src/core/CLucene/index/DocumentsWriter.cpp:472
(gdb) frame 5
#5  0x001e9e6b in lucene::index::DocumentsWriter::writeSegment
(this=0x45f5b430, flushedFiles=@0x45f5b55c) at
/Users/terylt/Projects/Canaris2/libs/clucene/src/core/CLucene/index/DocumentsWriter.cpp:595
(gdb) frame 4
#4  0x001e58c4 in lucene::index::DocumentsWriter::appendPostings
(this=0x45f5b430, fields=0xbfff9394, termsOut=0x53eeb3e0,
freqOut=0x437b65d0, proxOut=0x437b65f0) at
/Users/terylt/Projects/Canaris2/libs/clucene/src/core/CLucene/index/DocumentsWriter.cpp:760
(gdb) frame 3
#3  0x932283db in __assert_rtn ()
(gdb)


Basically, what I'm doing is building a set (a few hundred) of small search
engine files each 250 MBs in size and having 500,000 documents.    I build
the indexes one at a time using code similar to the demo:


*********************************Create Writer:

memset(output, 0, 2048);
strcpy(output, tempdir);
char* temp = mkdtemp(output)
if(temp == NULL)
{
      printf("Can't create temp indice file");
      exit(1);
}
m_writer = _CLNEW IndexWriter( output ,&an, true);
m_writer->setMaxFieldLength(0x7FFFFFFFL); // LUCENE_INT32_MAX_SHOULDBE
m_writer->setRAMBufferSizeMB(250.0);
m_writer->setMaxBufferedDocs(PARTITION_SIZE);
// Turn this off to make indexing faster; we'll turn it on later before
optimizing
m_writer->setUseCompoundFile(false);

**************************************Add document 500,000 times:

sub->m_writer->addDocument(&(conn->m_doc), &(sub->an));


******************************************Then I clean up:

sub->m_writer->setUseCompoundFile(true);
sub->m_writer->optimize();

// Close and clean up
sub->m_writer->close();

_CLLDELETE(sub->m_writer);


And then I recreate the next writer.    Everything works great, output is
exactly how I want it and everything.    The problem is that I get the above
assertional error at random times.  It doesn't seem to be attached to any in
particular input to the search engine, because if I run it again on the same
files, it won't happen on that file.    There seems to be a bit of an issue
with the Internal RAM Buffer.   Memory usage for the process seems to be
good up until the assertional error.

Anyone have any idea what might be wrong or what I might be able to do to
get a better understanding of the problem?

Any ideas would be great.

Best Regards,


Teryl