Hi all,

We are playing around with RC2 and are trying to get the spellcheck stuff running.  I noticed in a previous message to vufind-tech it was stated that you have to optimize your solr index before spellchecking worked.  Sure enough this did work for us on a limited sample of 25K records but I was wondering if someone could help shed some light on the errors that we are seeing when we index.  During the import after commit and before the import script exits it connects to solr and appears to try to build the spellchecker index (doesn't work).  After I ran the optimize.php command in the util folder our spellchecker worked but I was wondering if this is working as it should or if we should try to resolve the error that is coming up in the import process.

Thanks for any feedback,

Erik

(error from import process)

INFO [main] (MarcImporter.java:383) - Setting Solr closed flag
 INFO [main] (MarcImporter.java:409) - Connecting to solr server at URL: http://localhost:8080/solr/biblio/update
Jan 25, 2010 3:54:27 PM org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener buildSpellIndex
SEVERE: Exception in building spell check index for spellchecker: default
java.lang.RuntimeException: after flush: fdx size mismatch: 32365 docs vs 131072 length in bytes of _7.fdx file exists?=true
    at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
    at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:50)
    at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:380)
    at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
    at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4294)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
    at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2492)
    at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2446)
    at org.apache.lucene.search.spell.SpellChecker.indexDictionary(SpellChecker.java:345)
    at org.apache.lucene.search.spell.SpellChecker.indexDictionary(SpellChecker.java:362)
    at org.apache.solr.spelling.IndexBasedSpellChecker.build(IndexBasedSpellChecker.java:89)
    at org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener.buildSpellIndex(SpellCheckComponent.java:373)
    at org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener.newSearcher(SpellCheckComponent.java:361)
    at org.apache.solr.core.SolrCore$4.call(SolrCore.java:1163)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Jan 25, 2010 3:54:33 PM org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener buildSpellIndex
SEVERE: Exception in building spell check index for spellchecker: basicSpell
java.lang.RuntimeException: after flush: fdx size mismatch: 41453 docs vs 163840 length in bytes of _4.fdx file exists?=true
    at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
    at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:50)
    at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:380)
    at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
    at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4294)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
    at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2492)
    at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2446)
    at org.apache.lucene.search.spell.SpellChecker.indexDictionary(SpellChecker.java:345)
    at org.apache.lucene.search.spell.SpellChecker.indexDictionary(SpellChecker.java:362)
    at org.apache.solr.spelling.IndexBasedSpellChecker.build(IndexBasedSpellChecker.java:89)
    at org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener.buildSpellIndex(SpellCheckComponent.java:373)
    at org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener.newSearcher(SpellCheckComponent.java:361)
    at org.apache.solr.core.SolrCore$4.call(SolrCore.java:1163)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
 INFO [main] (SolrUpdate.java:119) - <?xml version="1.0" encoding="UTF-8"?>
 INFO [main] (SolrUpdate.java:119) - <response>
 INFO [main] (SolrUpdate.java:119) - <lst name="responseHeader"><int name="status">0</int><int name="QTime">16611</int></lst>
 INFO [main] (SolrUpdate.java:119) - </response>
 INFO [main] (MarcImporter.java:504) - Finished indexing in 4:32.00
 INFO [main] (MarcImporter.java:513) - Indexed 24890 at a rate of about 91.0 per sec
 INFO [main] (MarcImporter.java:514) - Deleted 0 records
 INFO [Thread-1] (MarcImporter.java:443) - Starting Shutdown hook
 INFO [Thread-1] (MarcImporter.java:462) - Finished Shutdown hoo

--
Erik Mitchell, Ph.D.
Assistant Director for Technology Services
Z. Smith Reynolds Library
Wake Forest University
http://erikmitchell.info