#151 hfst-ospell gets stuck on some platforms

[Flammie Pirinen]

Gdb runs out of memory, valgrind sometimes, works nicely on x86 laptop, results from hfst.ling.helsinki.fi. Test suite:

$ make check
make  speller.zhfst spl.hfstol sug.hfstol err.hfstol test.strings
make[1]: Siirrytään hakemistoon "/home/tapirine/src/hfst-svn-trunk/hfst-ospell"
make[1]: "speller.zhfst" on ajan tasalla.
make[1]: "spl.hfstol" on ajan tasalla.
make[1]: "sug.hfstol" on ajan tasalla.
make[1]: "err.hfstol" on ajan tasalla.
make[1]: Kohteelle "test.strings" ei tarvitse tehdä mitään.
make[1]: Poistutaan hakemistosta "/home/tapirine/src/hfst-svn-trunk/hfst-ospell"
make  check-TESTS
make[1]: Siirrytään hakemistoon "/home/tapirine/src/hfst-svn-trunk/hfst-ospell"
./basic-zhfst.sh: rivi 9:  4524 Valmis                  cat $srcdir/test.strings
  4525 Muistialueen ylitys     | ./hfst-ospell speller.zhfst
FAIL: basic-zhfst.sh
....
======================================
3 of 3 tests failed
Please report to hfst-bugs@helsinki.fi
======================================
make[1]: *** [check-TESTS] Virhe 1
make: *** [check-am] Keskeytys

Hand written string exits neatly in valgrind but not in shell:

==4374== Memcheck, a memory error detector
==4374== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==4374== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==4374== Command: /home/tapirine/src/hfst-ospell/.libs/lt-hfst-ospell speller.zhfst
==4374== 
olut
==4374== Invalid read of size 2
==4374==    at 0x4E40025: hfst_ol::Transducer::has_epsilons_or_flags(unsigned int) (ospell.cc:366)
==4374==    by 0x4E424E6: hfst_ol::Speller::lexicon_epsilons() (ospell.cc:181)
==4374==    by 0x4E435BB: hfst_ol::Speller::check(char*) (ospell.cc:505)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x8703728 is 200 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 2
==4374==    at 0x4E3FFE8: hfst_ol::Transducer::has_transitions(unsigned int, unsigned short) const (ospell.cc:356)
==4374==    by 0x4E41BDB: hfst_ol::Speller::lexicon_consume() (ospell.cc:212)
==4374==    by 0x4E435C3: hfst_ol::Speller::check(char*) (ospell.cc:506)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x8703734 is 212 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 4
==4374==    at 0x4E3FFBA: hfst_ol::Transducer::next(unsigned int, unsigned short) const (ospell.cc:346)
==4374==    by 0x4E41BF8: hfst_ol::Speller::lexicon_consume() (ospell.cc:217)
==4374==    by 0x4E435C3: hfst_ol::Speller::check(char*) (ospell.cc:506)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x8703736 is 214 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 2
==4374==    at 0x4E401D1: hfst_ol::Transducer::take_non_epsilons(unsigned int, unsigned short) const (ospell.cc:394)
==4374==    by 0x4E41C10: hfst_ol::Speller::lexicon_consume() (ospell.cc:219)
==4374==    by 0x4E435C3: hfst_ol::Speller::check(char*) (ospell.cc:506)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x870377c is 284 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 4
==4374==    at 0x4E40200: hfst_ol::Transducer::take_non_epsilons(unsigned int, unsigned short) const (hfst-ol.h:582)
==4374==    by 0x4E41C10: hfst_ol::Speller::lexicon_consume() (ospell.cc:219)
==4374==    by 0x4E435C3: hfst_ol::Speller::check(char*) (ospell.cc:506)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x8703784 is 292 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 2
==4374==    at 0x4E40204: hfst_ol::Transducer::take_non_epsilons(unsigned int, unsigned short) const (hfst-ol.h:566)
==4374==    by 0x4E41C10: hfst_ol::Speller::lexicon_consume() (ospell.cc:219)
==4374==    by 0x4E435C3: hfst_ol::Speller::check(char*) (ospell.cc:506)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x870377e is 286 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 4
==4374==    at 0x4E40209: hfst_ol::Transducer::take_non_epsilons(unsigned int, unsigned short) const (hfst-ol.h:566)
==4374==    by 0x4E41C10: hfst_ol::Speller::lexicon_consume() (ospell.cc:219)
==4374==    by 0x4E435C3: hfst_ol::Speller::check(char*) (ospell.cc:506)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x8703780 is 288 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 2
==4374==    at 0x4E401D1: hfst_ol::Transducer::take_non_epsilons(unsigned int, unsigned short) const (ospell.cc:394)
==4374==    by 0x4E41DCB: hfst_ol::Speller::lexicon_consume() (ospell.cc:230)
==4374==    by 0x4E435C3: hfst_ol::Speller::check(char*) (ospell.cc:506)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x8703788 is 296 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 2
==4374==    at 0x4E40045: hfst_ol::Transducer::has_epsilons_or_flags(unsigned int) (hfst-ol.h:559)
==4374==    by 0x4E424E6: hfst_ol::Speller::lexicon_epsilons() (ospell.cc:181)
==4374==    by 0x4E435BB: hfst_ol::Speller::check(char*) (ospell.cc:505)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x87037ac is 332 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 2
==4374==    at 0x4E40005: hfst_ol::Transducer::has_transitions(unsigned int, unsigned short) const (ospell.cc:354)
==4374==    by 0x4E41BDB: hfst_ol::Speller::lexicon_consume() (ospell.cc:212)
==4374==    by 0x4E435C3: hfst_ol::Speller::check(char*) (ospell.cc:506)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x87037ac is 332 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 2
==4374==    at 0x4E40277: hfst_ol::Transducer::is_final(unsigned int) (hfst-ol.h:589)
==4374==    by 0x4E4373B: hfst_ol::Speller::check(char*) (ospell.cc:502)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x87037e8 is 392 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 2
==4374==    at 0x4E4027E: hfst_ol::Transducer::is_final(unsigned int) (hfst-ol.h:589)
==4374==    by 0x4E4373B: hfst_ol::Speller::check(char*) (ospell.cc:502)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x87037ea is 394 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
==4374== Invalid read of size 4
==4374==    at 0x4E40286: hfst_ol::Transducer::is_final(unsigned int) (ospell.cc:402)
==4374==    by 0x4E4373B: hfst_ol::Speller::check(char*) (ospell.cc:502)
==4374==    by 0x4E4A656: hfst_ol::ZHfstOspeller::spell(std::string const&) (ZHfstOspeller.cc:166)
==4374==    by 0x402DCD: zhfst_spell(char*) (main.cc:207)
==4374==    by 0x402794: main (main.cc:292)
==4374==  Address 0x87037ec is 396 bytes inside a block of size 428 free'd
==4374==    at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4374==    by 0x4E4BAE8: hfst_ol::ZHfstOspeller::read_zhfst(std::string const&) (ZHfstOspeller.cc:242)
==4374==    by 0x402C05: zhfst_spell(char*) (main.cc:161)
==4374==    by 0x402794: main (main.cc:292)
==4374== 
"olut" is in the lexicon

==4374== 
==4374== HEAP SUMMARY:
==4374==     in use at exit: 5,096 bytes in 81 blocks
==4374==   total heap usage: 973 allocs, 892 frees, 494,080 bytes allocated
==4374== 
==4374== LEAK SUMMARY:
==4374==    definitely lost: 2,274 bytes in 11 blocks
==4374==    indirectly lost: 1,856 bytes in 50 blocks
==4374==      possibly lost: 0 bytes in 0 blocks
==4374==    still reachable: 966 bytes in 20 blocks
==4374==         suppressed: 0 bytes in 0 blocks
==4374== Rerun with --leak-check=full to see details of leaked memory
==4374== 
==4374== For counts of detected and suppressed errors, rerun with: -v
==4374== ERROR SUMMARY: 32 errors from 13 contexts (suppressed: 40 from 26)

Using test suite:

$ cat test.strings | libtool --mode=execute valgrind hfst-ospell speller.zhfst
==4614== Memcheck, a memory error detector
==4614== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==4614== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==4614== Command: /home/tapirine/src/hfst-ospell/.libs/lt-hfst-ospell speller.zhfst
==4614== 
==4614== Invalid read of size 2
...
==4614== More than 10000000 total errors detected.  I'm not reporting any more.
==4614== Final error counts will be inaccurate.  Go fix your program!
==4614== Rerun with --error-limit=no to disable this cutoff.  Note
==4614== that errors may occur in your program without prior warning from
==4614== Valgrind, because errors are no longer being displayed.

Using real data:

$ cat corp/kl/wikipedia-test.1.nonwords | valgrind hfst-ospell-survey hfst/kl/errmodel.ed1unwe.hfst hfst/kl/acceptor.gtklprod.hfst 
==450== Memcheck, a memory error detector
==450== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==450== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==450== Command: hfst-ospell-survey hfst/kl/errmodel.ed1unwe.hfst hfst/kl/acceptor.gtklprod.hfst
==450== 
==450== Warning: set address range perms: large range [0x727bfb040, 0x73bbfb030) (undefined)
**450** new/new[] failed and should throw an exception, but Valgrind
**450**    cannot throw exceptions and so is aborting instead.  Sorry.

==450==     Valgrind's memory management: out of memory:
==450==        newSuperblock's request for 1009733632 bytes failed.
==450==        34193317888 bytes have already been allocated.
==450==     Valgrind cannot continue.  Sorry.

The version from few weeks back produced some results on real data unless it was old version for some reason.