#300 Conversion to foma format triggers segmentation fault

[sjurum]

To repeat:

$ hfst-fst2fst -b -F -i analyser-gt-desc.hfstol -o analyser-gt-desc.foma
Segmentation fault: 11
$ hfst-fst2fst -b -F -i analyser-gt-desc.hfst -o analyser-gt-desc.foma
Segmentation fault: 11

Removing the -b option does not help. Hfst version (svn revision 4374):

$ hfst-info
No tests selected; printing known data
HFST info version: 0.1
HFST packaging: hfst 3.8.2
HFST version: 3.8.2
HFST long version: 300080002
HFST configuration revision: $Revision: 4279 $
OpenFst supported
SFST supported
foma supported
Unicode support: glib

MacOS 10.10.3.

[sjurum]

I tried the same code on a MacOSX 10.6 system - no segmentation fault. MacOSX 10.6 uses an Apple version of GCC:

$ gcc --version
i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

MacOSX 10.10 uses LLVM/Clang:

$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.3.0
Thread model: posix

It might be that the segfault is caused by this compiler difference. And it might be because of something on my system. I'll ask a colleague with 10.10 to try to replicate the bug and report back.

[sjurum]

My colleague tested both on MacOSX 10.10 and on Linux:

1. 10.10: segfault
2. Linux: pass

His finding rules out that the problem is restricted to my computer, and strengthens the case that the crash is related to differences in the compilers: GCC (ok) vs LLVM (segfault). Exactly how and why I have no idea.

[Anonymous]

Another update from my colleague:

He has now rebuilt hfst3 on Linux using Clang/LLVM as follows:

CC=clang CXX=clang++ ./configure --with-unicode-handler=glib --enable-all-tools --enable-silent-rules && make && sudo make install
src $ clang -v 
Ubuntu clang version 3.6.0-2ubuntu1 (tags/RELEASE_360/final) (based on LLVM 3.6.0
) 
Target: x86_64-pc-linux-gnu 
Thread model: posix
[...]
 502  cd $GTHOME/langs/sme
 503  ./configure --with-hfst --without-xfst 
 504  make clean 
 505  make -j3 
 506  history |grep fst2 
 507  cd src/ 
 508  hfst-fst2fst -b -F analyser-gt-desc.hfst > analyser-gt-desc.foma

No errors, no crashes. That is, using Clang/LLVM on Linux, hfst builds fine, and conversion to foma format with backend-native wrapper is working exactly as it should.

This weakens the case that the bug is related to the compiler. What is left is a combo of the compiler and the OS, perhaps something with the linked C(++) library?

[sjurum]

I spent some time with Måns in June trying to identify the cause of this bug. Using a similar setup as mine, he was able to reproduce it.

The summary of the findings are:
1. it is related to the changes done to the foma code to make it fit with hfst (ie hfst-specific changes in foma)
2. it is triggered only on macosx (recent versions)

To minimize the differences between the two code bases, Måns included some of the hfst changes in his code base. He then made a new release. This new release has later been incorporated into hfst.

BUT: The bug is still there:

$ echo test | hfst-strings2fst | hfst-fst2fst -b -F
Segmentation fault: 11

This is using revision 4438 of hfst trunk.

[sjurum]

Any progress on this bug?