#299 Indri fails to compile on OS X Sierra

[Ian Soboroff]

I downloaded the 5.11 tarball and built it on OS X Sierra. Reading the documentation, I made a parameter file for TREC-8 (NIST’s motto: if it doesn’t work on TREC-8, walk away). I’ve attached it.

When I run IndriBuildIndex params.xml, I get a segfault:

moose:trec8-indri soboroff$ IndriBuildIndex params.xml
kstem_add_table_entry: Duplicate word emeritus will be ignored.
0:00: Created repository /Users/soboroff/lemur/trec8-indri/index
Adding text to trectext as an indexed field
Adding ti to trectext as an indexed field
Adding ti to trectext as an included tag
Adding f to trectext as an indexed field
Adding f to trectext as an included tag
Adding p to trectext as an indexed field
Adding p to trectext as an included tag
0:00: Opened /home/collections/TREC-Disk-4/FR94/04/FR940404.1
Segmentation fault: 11 (core dumped)

I corresponded with David Fisher, who confirmed:

Sierra hasn’t been tested. The may well be compilation issues that will need to be addressed (the first change to clang introduced a lot of problems). As the error is with respect to memory allocation (see below), I expect that it may be either libc++ or compiler related (indri has not transition to libc++, it still uses -libstdc++).

I have downloaded the same indri distribution and built it on my home machine, which is running High Sierra.

I am using just the FR94 dataset (repackaged from the cdrom ages ago, I do not have the original files).

I replicated your core dump.

If you didn’t, enable core dumps with ulimit -c unlimited.

The core will be dumped in /cores/core.<pid></pid>

You can use the Xcode debugger to examine the core: https://wincent.com/wiki/Debugging_core_dumps_on_OS_X

I got this:

earthsea:Development dfisher$ lldb -c /cores/core.9286
(lldb) target create --core "/cores/core.9286"
Core file '/cores/core.9286' (x86_64) was loaded.
(lldb) bt
IndriBuildIndex was compiled with optimization - stepping may behave oddly; variables may not be available.

• thread #1: tid = 0x0000, 0x0000000100784b11 IndriBuildIndex`indri::index::DocListMemoryBuilder::DocListMemoryBuilder(indri::utility::RegionAllocator*) [inlined] indri::index::DocListMemoryBuilder::DocListMemoryBuilder(this=0x0000000101def528, allocator=0x00007ffc9801a608) + 45 at DocListMemoryBuilder.cpp:35, stop reason = signal SIGSTOP
• frame #0: 0x0000000100784b11 IndriBuildIndexindri::index::DocListMemoryBuilder::DocListMemoryBuilder(indri::utility::RegionAllocator*) [inlined] indri::index::DocListMemoryBuilder::DocListMemoryBuilder(this=0x0000000101def528, allocator=0x00007ffc9801a608) + 45 at DocListMemoryBuilder.cpp:35 [opt] frame #1: 0x0000000100784ae4 IndriBuildIndexindri::index::DocListMemoryBuilder::DocListMemoryBuilder(this=0x0000000101def528, allocator=0x00007ffc9801a608) + 4 at DocListMemoryBuilder.cpp:45 [opt]
  frame #2: 0x00000001007a7271 IndriBuildIndexindri::index::MemoryIndex::_lookupTerm(char const*) [inlined] indri::index::MemoryIndex::term_entry::term_entry(allocator=<unavailable>) + 673 at MemoryIndex.hpp:57 [opt] frame #3: 0x00000001007a725d IndriBuildIndexindri::index::MemoryIndex::_lookupTerm(char const*) [inlined] indri::index::MemoryIndex::term_entry::term_entry(allocator=<unavailable>) at MemoryIndex.hpp:59 [opt]
  frame #4: 0x00000001007a725d IndriBuildIndexindri::index::MemoryIndex::_lookupTerm(this=<unavailable>, term=<unavailable>) + 653 at MemoryIndex.cpp:509 [opt] frame #5: 0x00000001007a7564 IndriBuildIndexindri::index::MemoryIndex::addDocument(this=<unavailable>, document=<unavailable>) + 500 at MemoryIndex.cpp:576 [opt]
  frame #6: 0x00000001007c19d3 IndriBuildIndexindri::collection::Repository::addDocument(this=<unavailable>, document=<unavailable>, inCollection=<unavailable>) + 403 at Repository.cpp:542 [opt] frame #7: 0x0000000100790e3c IndriBuildIndexindri::api::IndexEnvironment::addFile(this=<unavailable>, fileName=<unavailable>, fileClass=<unavailable>) + 988 at IndexEnvironment.cpp:424 [opt]
  frame #8: 0x0000000100766e95 IndriBuildIndexmain(argc=<unavailable>, argv=<unavailable>) + 7045 at IndriBuildIndex.cpp:1099 [opt] frame #9: 0x00007fff6a759115 libdyld.dylibstart + 1
  frame #10: 0x00007fff6a759115 libdyld.dylib`start + 1
  (lldb) </unavailable></unavailable></unavailable></unavailable></unavailable></unavailable>

Reducing to the first document of the FR set produces the fault, whereas a single hand edited test document in trecformat does not.

The problem appears to occur after three or more terms are encountered.

[David Fisher]

Compiling without optimization (-O0) allows indexing of the FR94 data:

earthsea:Development dfisher$ indri-5.11/buildindex/IndriBuildIndex params.xml
kstem_add_table_entry: Duplicate word emeritus will be ignored.
0:00: Created repository fr94-index
Adding text to trectext as an indexed field
Adding ti to trectext as an indexed field
Adding ti to trectext as an included tag
Adding f to trectext as an indexed field
Adding f to trectext as an included tag
Adding p to trectext as an indexed field
Adding p to trectext as an included tag
0:00: Opened fr94.dat
1:09: Documents parsed: 55630 Documents indexed: 55630
1:09: Closed fr94.dat
1:09: Closing index
1:22: Finished
earthsea:Development dfisher$

That should help isolate the cause.

[J Mac]

Hi there,

I had similar issues compiling on the latest version of Ubuntu and gcc 7.2.
After seeing a bug similar to this before (what seems to be an over-optimization issue), I recompiled using the additional flag -"fno-tree-vectorize" and have had no problems since.

Maybe Ian could try this and see if it resolves the issue?

[Wei Zhong]

Hi J Mac, thank you for posting this info, you have saved me the day! I can confirm newer gcc has the problem (not sure starting from which version, but my workstation at work which is Ubuntu works fine but my home PC which is ArchLinux gets segementation fault). And by adding the -fno-tree-vectorize CFLAG, the segementation fault is gone.