#90 Export of dataabases crashes

[Stefan Meyer]

Trying to export an .sci database fails with message:

0
(func) exportGames
(file) db_database.cpp:1383
(what) precondition violation: illegalRejected || !format::isScidFormat(destination.format())
(type) mstl::precondition_violation_exception

=== Backtrace ============================================
db::Database::exportGames<db::Consumer> [db_database.cpp:1383]
app::View::exportGames [app_view.cpp:667]
app::View::exportGames(mstl::string const&, mstl::string const&, mstl::string const&, unsigned int, db::type::ID, unsigned int, mstl::bitfield<unsigned __int128=""> const&, bool, mstl::vector<mstl::string> const, unsigned int, unsigned int, db::Log&, util::Progress&, app::View::FileMode [app_view.cpp:772]
cmdExport [tcl_view.cpp:695]
safeCall [tcl_base.cpp:764]
main [tkscidb.cpp:109]
==========================================================

while executing

"::util::catchException $cmd count "
(procedure "DoExport" line 262)
invoked from within
"DoExport .application.nb.database.main.switcher.content .application.nb.database.main.switcher.content.export /home/stefan/privat/schach/export/TestEx..."
(in namespace inscope "::export" script line 1)
invoked from within
"{*}$Vars(okcommand) $files"
(procedure "Activate" line 134)
invoked from within
"Activate .application.nb.database.main.switcher.content.export.top.nb.filedialog.fsbox yes"
(in namespace inscope "::fsbox" script line 1)
invoked from within
"::namespace inscope ::fsbox {Activate .application.nb.database.main.switcher.content.export.top.nb.filedialog.fsbox yes}"
invoked from within
".application.nb.database.main.switcher.content.export.top.nb.filedialog.fsbox.buttons.ok invoke "
invoked from within
".application.nb.database.main.switcher.content.export.top.nb.filedialog.fsbox.buttons.ok instate !disabled { .application.nb.database.main.switcher...."
invoked from within
".application.nb.database.main.switcher.content.export.top.nb.__filedialog.fsbox.buttons.ok instate pressed { .application.nb.database.main.switcher.co..."
(command bound to event)

I use revision 1089 on debian lenny

Steps to reproduce:

1) Create new sci database
2) Save one game (nothing special only starting position)
3) Choose export in database view
4) Choose SCID as export format in left "frame"
=> Export fails

[Gregor Cramer]

Thanks for your error report. In fact it was an error in debugging code, fixed in rev. 1090.

[Stefan Meyer]

Thanks for the fast fix.

Some of my databases now create the following error, when trying to export:

0
(func) adjustListSize
(file) si3/si3_name_list.cpp:245
(what) assertion failed: m_list.size() == size()
(type) mstl::assertion_failure_exception

=== Backtrace ============================================
db::si3::NameList::adjustListSize [si3/si3_name_list.cpp:245]
db::si3::NameList::update [si3/si3_name_list.cpp:151]
db::si3::Codec::writeNamebases [si3/si3_codec.cpp:2063]
db::si3::Codec::writeNamebases [si3/si3_codec.cpp:2037]
db::si3::Codec::save [si3/si3_codec.cpp:776]
db::si3::Codec::save [si3/si3_codec.cpp:784]
db::DatabaseCodec::save [db_database_codec.ipp:62]
db::Database::save [db_database.cpp:458]
app::View::exportGames(mstl::string const&, mstl::string const&, mstl::string const&, unsigned int, db::type::ID, unsigned int, mstl::bitfield<unsigned __int128=""> const&, bool, mstl::vector<mstl::string> const, unsigned int, unsigned int, db::Log&, util::Progress&, app::View::FileMode [app_view.cpp:777]
cmdExport [tcl_view.cpp:695]
safeCall [tcl_base.cpp:764]
main [tkscidb.cpp:109]
==========================================================

while executing

"::util::catchException $cmd count "
(procedure "DoExport" line 262)
invoked from within
"DoExport .application.nb.database.main.switcher.content .application.nb.database.main.switcher.content.export {/home/stefan/privat/schach/export/Aljec..."
(in namespace inscope "::export" script line 1)
invoked from within
"{*}$Vars(okcommand) $files"
(procedure "Activate" line 134)
invoked from within
"Activate .application.nb.database.main.switcher.content.export.top.nb.filedialog.fsbox yes"
(in namespace inscope "::fsbox" script line 1)
invoked from within
"::namespace inscope ::fsbox {Activate .application.nb.database.main.switcher.content.export.top.nb.filedialog.fsbox yes}"
invoked from within
".application.nb.database.main.switcher.content.export.top.nb.filedialog.fsbox.buttons.ok invoke "
invoked from within
".application.nb.database.main.switcher.content.export.top.nb.filedialog.fsbox.buttons.ok instate !disabled { .application.nb.database.main.switcher...."
invoked from within
".application.nb.database.main.switcher.content.export.top.nb.__filedialog.fsbox.buttons.ok instate pressed { .application.nb.database.main.switcher.co..."
(command bound to event)

Please let me know, if you need an exanple database

[Gregor Cramer]

I've sent an email to your users.sf.net account for upload request of an example database.

[Stefan Meyer]

I 've uploaded an scidb archive.

[Gregor Cramer]

Thanks for the database "Aljechin Alarm", unfortunately the export to .si4 is working properly. What I have done:

• scidb-beta "Aljechin Alarm.scv"
• open Export on database tile
• choosing Scid format
• saving the file (this is doing the export)

No error at all, and opening the .si4 file works properly. Probably you have send me the wrong database?

[Stefan Meyer]

No, still reproducable for me.
I tried the same steps and uploaded the Aljechin Alarm.scv and the partly genrated Aljechin Alarm.si4.

How can I help you?

[Gregor Cramer]

Still the same, I have no problems with export. Probably your built failed, please try the follwing:

• cd ~/scidb-code # change to the installation directory
• make clean
• svn update
• make
• sudo make install

and try again to export. And if this does not help, then please try another step:

• cd ~ # change to your home directory
• mv .scidb-beta .scidb-beta.bak
• scidb-beta # and try again to export
• mv .scidb-beta.bak .scidb-beta

If this succeeds, then something in the configuration file is wrong, please send me your configuration file ~/.scidb-beta/config/options.dat in this case.

[Stefan Meyer]

I tried both, no change, still the same error.

[Gregor Cramer]

Many thanks for your help. This is really mysterious. The only idea I have is to test your executable on my machine. But this is possible only if the dependencies will be fulfilled. Please paste the output of

ldd `which tkscidb-beta`

in the next post. About my machine: Ubuntu 14.04.1 LTS (trusty), 32 bit processor.

[Stefan Meyer]

bash>lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.5 (jessie)
Release:        8.5
Codename:       jessie

64-bit

bash>ldd `which tkscidb-beta`
 ldd `which tkscidb-beta`
        linux-vdso.so.1 (0x00007fff3a068000)
        libtk8.6.so => /usr/lib/x86_64-linux-gnu/libtk8.6.so (0x00007f6d1e636000)
        libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f6d1e2f3000)
        libSM.so.6 => /usr/lib/x86_64-linux-gnu/libSM.so.6 (0x00007f6d1e0eb000)
        libICE.so.6 => /usr/lib/x86_64-linux-gnu/libICE.so.6 (0x00007f6d1dece000)
        libXcursor.so.1 => /usr/lib/x86_64-linux-gnu/libXcursor.so.1 (0x00007f6d1dcc3000)
        libtcl8.6.so => /usr/lib/x86_64-linux-gnu/libtcl8.6.so (0x00007f6d1d90c000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6d1d6f1000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f6d1d4c8000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6d1d2ab000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f6d1cfa0000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6d1cc9f000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6d1ca89000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6d1c6de000)
        libXft.so.2 => /usr/lib/x86_64-linux-gnu/libXft.so.2 (0x00007f6d1c4c8000)
        libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007f6d1c28b000)
        libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f6d1bfe1000)
        libXss.so.1 => /usr/lib/x86_64-linux-gnu/libXss.so.1 (0x00007f6d1bdde000)
        libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f6d1bbcc000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6d1b9c8000)
        libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f6d1b7a6000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f6d1b5a1000)
        libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f6d1b397000)
        libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007f6d1b191000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6d1e99a000)
        libpng12.so.0 => /usr/lib/x86_64-linux-gnu/libpng12.so.0 (0x00007f6d1af6a000)
        libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f6d1ad66000)
        libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f6d1ab61000)

I will try it tomorrow on another machine with ubuntu 15.10 32 bit.

[Gregor Cramer]

I guess it works on your 32 bit machine. Now I think I know the problem, it only occurs on 64 bit machines, for any reason this part is not 64 bit safe. Unluckily I don't have a 64 bit machine, so I have to think about how I find the hole, with your help.

[Gregor Cramer]

I think a first test with valgrind is the best solution. Please install valgrind:

sudo apt-get install valgrind

and start Scidb in the following way:

valgrind /usr/local/bin/tkscidb-beta /usr/local/bin/scidb-beta > /tmp/log

and attach the log file to the post. Valgrind will detect memory problems, if any.

[Stefan Meyer]

Done.
Hier die Ausgabe:

Failed to read a valid object file image from memory.
147     m_syswrap/syscall-amd64-linux.S: Datei oder Verzeichnis nicht gefunden.
Failed to read a valid object file image from memory.
147     m_syswrap/syscall-amd64-linux.S: Datei oder Verzeichnis nicht gefunden.
==31007== Conditional jump or move depends on uninitialised value(s)
==31007==    at 0x4C30083: strstr (vg_replace_strmem.c:1452)
==31007==    by 0x766ACB: mstl::backtrace::text_write(mstl::ostringstream&, unsigned int) const (m_backtrace.cpp:844)
==31007==    by 0x76E38A: mstl::bits::prepare_msg(mstl::exception&, char const*, unsigned int, char const*, char const*) (m_exception.cpp:132)
==31007==    by 0x76E487: mstl::bits::prepare_exc(mstl::exception&, char const*, unsigned int, char const*, char const*) (m_exception.cpp:158)
==31007==    by 0x73551B: void mstl::bits::throw_exc<mstl::assertion_failure_exception>(mstl::assertion_failure_exception const&, char const*, int, char const*) (m_exception.ipp:51)
==31007==    by 0x73479E: db::si3::NameList::adjustListSize() (si3_name_list.cpp:245)
==31007==    by 0x73412D: db::si3::NameList::update(db::Namebase&, sys::utf8::Codec&) (si3_name_list.cpp:151)
==31007==    by 0x724022: db::si3::Codec::writeNamebases(mstl::ostream&, util::Progress*) (si3_codec.cpp:2063)
==31007==    by 0x723E4C: db::si3::Codec::writeNamebases(mstl::string const&) (si3_codec.cpp:2037)
==31007==    by 0x71D536: db::si3::Codec::save(mstl::string const&, unsigned int, util::Progress&, bool) (si3_codec.cpp:776)
==31007==    by 0x71D685: db::si3::Codec::save(mstl::string const&, unsigned int, util::Progress&) (si3_codec.cpp:784)
==31007==    by 0x5F17EF: db::DatabaseCodec::save(unsigned int, util::Progress&) (db_database_codec.ipp:62)
==31007== 
==31007== Conditional jump or move depends on uninitialised value(s)
==31007==    at 0x4C30083: strstr (vg_replace_strmem.c:1452)
==31007==    by 0x766AE5: mstl::backtrace::text_write(mstl::ostringstream&, unsigned int) const (m_backtrace.cpp:845)
==31007==    by 0x76E38A: mstl::bits::prepare_msg(mstl::exception&, char const*, unsigned int, char const*, char const*) (m_exception.cpp:132)
==31007==    by 0x76E487: mstl::bits::prepare_exc(mstl::exception&, char const*, unsigned int, char const*, char const*) (m_exception.cpp:158)
==31007==    by 0x73551B: void mstl::bits::throw_exc<mstl::assertion_failure_exception>(mstl::assertion_failure_exception const&, char const*, int, char const*) (m_exception.ipp:51)
==31007==    by 0x73479E: db::si3::NameList::adjustListSize() (si3_name_list.cpp:245)
==31007==    by 0x73412D: db::si3::NameList::update(db::Namebase&, sys::utf8::Codec&) (si3_name_list.cpp:151)
==31007==    by 0x724022: db::si3::Codec::writeNamebases(mstl::ostream&, util::Progress*) (si3_codec.cpp:2063)
==31007==    by 0x723E4C: db::si3::Codec::writeNamebases(mstl::string const&) (si3_codec.cpp:2037)
==31007==    by 0x71D536: db::si3::Codec::save(mstl::string const&, unsigned int, util::Progress&, bool) (si3_codec.cpp:776)
==31007==    by 0x71D685: db::si3::Codec::save(mstl::string const&, unsigned int, util::Progress&) (si3_codec.cpp:784)
==31007==    by 0x5F17EF: db::DatabaseCodec::save(unsigned int, util::Progress&) (db_database_codec.ipp:62)
==31007== 
==31007== Conditional jump or move depends on uninitialised value(s)
==31007==    at 0x4C30083: strstr (vg_replace_strmem.c:1452)
==31007==    by 0x766AFF: mstl::backtrace::text_write(mstl::ostringstream&, unsigned int) const (m_backtrace.cpp:846)
==31007==    by 0x76E38A: mstl::bits::prepare_msg(mstl::exception&, char const*, unsigned int, char const*, char const*) (m_exception.cpp:132)
==31007==    by 0x76E487: mstl::bits::prepare_exc(mstl::exception&, char const*, unsigned int, char const*, char const*) (m_exception.cpp:158)
==31007==    by 0x73551B: void mstl::bits::throw_exc<mstl::assertion_failure_exception>(mstl::assertion_failure_exception const&, char const*, int, char const*) (m_exception.ipp:51)
==31007==    by 0x73479E: db::si3::NameList::adjustListSize() (si3_name_list.cpp:245)
==31007==    by 0x73412D: db::si3::NameList::update(db::Namebase&, sys::utf8::Codec&) (si3_name_list.cpp:151)
==31007==    by 0x724022: db::si3::Codec::writeNamebases(mstl::ostream&, util::Progress*) (si3_codec.cpp:2063)
==31007==    by 0x723E4C: db::si3::Codec::writeNamebases(mstl::string const&) (si3_codec.cpp:2037)
==31007==    by 0x71D536: db::si3::Codec::save(mstl::string const&, unsigned int, util::Progress&, bool) (si3_codec.cpp:776)
==31007==    by 0x71D685: db::si3::Codec::save(mstl::string const&, unsigned int, util::Progress&) (si3_codec.cpp:784)
==31007==    by 0x5F17EF: db::DatabaseCodec::save(unsigned int, util::Progress&) (db_database_codec.ipp:62)
==31007== 
Failed to read a valid object file image from memory.
147     m_syswrap/syscall-amd64-linux.S: Datei oder Verzeichnis nicht gefunden.

[Stefan Meyer]

After closing the valgrind session the following summary is shown:

==30878== 
==30878== HEAP SUMMARY:
==30878==     in use at exit: 68,830,116 bytes in 266,295 blocks
==30878==   total heap usage: 518,377 allocs, 252,082 frees, 93,784,581 bytes allocated
==30878== 
==30878== LEAK SUMMARY:
==30878==    definitely lost: 336,096 bytes in 57 blocks
==30878==    indirectly lost: 669,780 bytes in 593 blocks
==30878==      possibly lost: 16,952,718 bytes in 371 blocks
==30878==    still reachable: 50,871,522 bytes in 265,274 blocks
==30878==         suppressed: 0 bytes in 0 blocks
==30878== Rerun with --leak-check=full to see details of leaked memory
==30878== 
==30878== For counts of detected and suppressed errors, rerun with: -v
==30878== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 1 from 1)

I made a new run ----leak-check=full ...
Output is attached

[Gregor Cramer]

Many thanks, unfortunately valgrind is not reporting anything relevant. But after I've studied the code again I guessed that the bitset implementation can be the reason. So I tried a 64 bit emulation of the bitset, and now I can reproduce the problem! Soon I will release a fix.

[Gregor Cramer]

I've released v1091, this version works (with your example database). It was indeed a 64 bit problem, in one 64 bit wide variable onle the lower 32 bit were set. Thanks for your patience!

[Stefan Meyer]

Works perfect, thanks for the quick bugfix!