#247 merge index fails for indexes with a corpus

[Michael Zarozinski]

When performing a "merge-index", if the indexes have a corpus the merge fails with the following error:

java -jar homer-0.4-SNAPSHOT.jar merge-index --inputPath+/mnt/nfs/work2/michaelz/indexes/robust-index --inputPath+/mnt/nfs/
work2/michaelz/indexes/book-index --indexPath=/mnt/nfs/work2/michaelz/indexes/merged-test
java.class.path: homer-0.4-SNAPSHOT.jar
cwd: /mnt/nfs/work2/michaelz/Proteus/homer/target/.
2 indexes found.
Dec 01, 2014 11:32:30 AM org.lemurproject.galago.core.index.merge.MergeIndex getJob
INFO: Merging Part: postings
Dec 01, 2014 11:32:30 AM org.lemurproject.galago.core.index.merge.MergeIndex getJob
INFO: Merging Part: postings.krovetz
Dec 01, 2014 11:32:30 AM org.lemurproject.galago.core.index.merge.MergeIndex getJob
INFO: Merging Part: corpus
Dec 01, 2014 11:32:30 AM org.lemurproject.galago.core.index.merge.MergeIndex getJob
INFO: Merging Part: names
Dec 01, 2014 11:32:30 AM org.lemurproject.galago.core.index.merge.MergeIndex getJob
INFO: Merging Part: lengths
Dec 01, 2014 11:32:30 AM org.lemurproject.galago.core.index.merge.MergeIndex getJob
INFO: Merging Part: names.reverse
Created executor: org.lemurproject.galago.tupleflow.execution.LocalCheckpointedStageExecutor@398326be
Running without server!
Use --server=true to enable web-based status page.
Stage indexNumberer completed with 0 errors.
Stage documentMapper completed with 0 errors.
Stage lengthsMergeStage completed with 0 errors.
Stage names.reverseMergeStage completed with 0 errors.
Stage namesMergeStage completed with 0 errors.
Dec 01, 2014 11:35:10 AM org.lemurproject.galago.core.index.merge.IndexPartMergeManager close
SEVERE: Errored Merging Part: corpus
java.io.IOException: Key, are out of order.
at org.lemurproject.galago.core.index.disk.DiskBTreeWriter.add(DiskBTreeWriter.java:106)
at org.lemurproject.galago.core.index.corpus.CorpusFileWriter.process(CorpusFileWriter.java:51)
at org.lemurproject.galago.core.index.corpus.CorpusFileWriter.process(CorpusFileWriter.java:27)
at org.lemurproject.galago.core.index.merge.CorpusMerger.performValueMerge(CorpusMerger.java:38)
at org.lemurproject.galago.core.index.merge.GenericIndexMerger.performKeyMerge(GenericIndexMerger.java:66)
at org.lemurproject.galago.core.index.merge.IndexPartMergeManager.close(IndexPartMergeManager.java:90)
at org.lemurproject.galago.core.types.DocumentSplit$FileIdOrder$TupleUnshredder.close(DocumentSplit.java:1992)
at org.lemurproject.galago.core.types.DocumentSplit$FileIdOrder$DuplicateEliminator.close(DocumentSplit.java:1950)
at org.lemurproject.galago.core.types.DocumentSplit$FileIdOrder$ShreddedReader.run(DocumentSplit.java:1899)
at org.lemurproject.galago.tupleflow.FileOrderedReader.run(FileOrderedReader.java:76)
at org.lemurproject.galago.tupleflow.execution.LocalCheckpointedStageExecutor$LocalExecutionStatus.run(LocalCheckpointedStageExecutor.java:96)
at java.lang.Thread.run(Thread.java:745)

java.io.IOException: java.io.IOException: Key, are out of order.
at org.lemurproject.galago.core.index.merge.IndexPartMergeManager.close(IndexPartMergeManager.java:95)
at org.lemurproject.galago.core.types.DocumentSplit$FileIdOrder$TupleUnshredder.close(DocumentSplit.java:1992)
at org.lemurproject.galago.core.types.DocumentSplit$FileIdOrder$DuplicateEliminator.close(DocumentSplit.java:1950)
at org.lemurproject.galago.core.types.DocumentSplit$FileIdOrder$ShreddedReader.run(DocumentSplit.java:1899)
at org.lemurproject.galago.tupleflow.FileOrderedReader.run(FileOrderedReader.java:76)
at org.lemurproject.galago.tupleflow.execution.LocalCheckpointedStageExecutor$LocalExecutionStatus.run(LocalCheckpointedStageExecutor.java:96)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Key, are out of order.
at org.lemurproject.galago.core.index.disk.DiskBTreeWriter.add(DiskBTreeWriter.java:106)
at org.lemurproject.galago.core.index.corpus.CorpusFileWriter.process(CorpusFileWriter.java:51)
at org.lemurproject.galago.core.index.corpus.CorpusFileWriter.process(CorpusFileWriter.java:27)
at org.lemurproject.galago.core.index.merge.CorpusMerger.performValueMerge(CorpusMerger.java:38)
at org.lemurproject.galago.core.index.merge.GenericIndexMerger.performKeyMerge(GenericIndexMerger.java:66)
at org.lemurproject.galago.core.index.merge.IndexPartMergeManager.close(IndexPartMergeManager.java:90)
... 6 more
Stage corpusMergeStage completed with 1 errors.
java.lang.Exception: java.io.IOException: java.io.IOException: Key, are out of order.
Exception in thread "main" java.util.concurrent.ExecutionException: Stage threw an exception:
at org.lemurproject.galago.tupleflow.execution.JobExecutor$JobExecutionStatus.waitForStages(JobExecutor.java:1051)
at org.lemurproject.galago.tupleflow.execution.JobExecutor$JobExecutionStatus.run(JobExecutor.java:981)
at org.lemurproject.galago.tupleflow.execution.JobExecutor.runWithoutServer(JobExecutor.java:1115)
at org.lemurproject.galago.tupleflow.execution.JobExecutor.runLocally(JobExecutor.java:1161)
at org.lemurproject.galago.core.tools.AppFunction.runTupleFlowJob(AppFunction.java:101)
at org.lemurproject.galago.core.index.merge.MergeIndex.run(MergeIndex.java:158)
at org.lemurproject.galago.core.tools.AppFunction.run(AppFunction.java:55)
at org.lemurproject.galago.core.tools.App.run(App.java:82)
at org.lemurproject.galago.core.tools.App.run(App.java:73)
at ciir.proteus.build.Main.main(Main.java:23)
Caused by: java.lang.Exception: java.io.IOException: java.io.IOException: Key, are out of order.
at org.lemurproject.galago.tupleflow.execution.LocalCheckpointedStageExecutor$LocalExecutionStatus.run(LocalCheckpointedStageExecutor.java:99)
at java.lang.Thread.run(Thread.java:745)

[Michael Zarozinski]

fixed: https://sourceforge.net/p/lemur/galago/ci/d5d7447061cb30f6daeeca2365d73c84cf5b8c0a/