#13 Multitasking crash

[Fredrik]

I rather commonly get crashes where a tread hangs but doesn't kill. Most oftenly, they are due to a double free somewhere in OBSmartsPattern::Match, and as far as I can tell, they appear when two different users/clients are accessing Mychem functionality at the same time.

I can reproduce the error by running SELECT MOLWEIGHT(compound_mol.mol) FROM compound_mol; on a db with ~10000 compounds on two different clients so that the second one starts before the first one ends, i.e. follow the following steps:
1. Connect to db with two different clients/sessions
2. run SELECT MOLWEIGHT(compound_mol.mol) FROM compound_mol; in first client
3. switch to second client
4. run SELECT MOLWEIGHT(compound_mol.mol) FROM compound_mol; again
5. check at the mysql error log and see crash info

An example of my crash log is below.

Kind regards,
Fredrik

More info:

mysql> select version(), mychem_version(), openbabel_version();
+---------------------+------------------+---------------------+
| version() | mychem_version() | openbabel_version() |
+---------------------+------------------+---------------------+
| 5.1.41-3ubuntu12.10 | 0.8.2 | 2.2.3 |
+---------------------+------------------+---------------------+
(But I have the same problem with mychem 0.7.0 and 0.8.1)

Excerpts from error.log:

*** glibc detected *** /usr/sbin/mysqld: double free or corruption (fasttop): 0x00007f545410b5b0 ***
======= Backtrace: =========
/lib/libc.so.6(+0x775b6)[0x7f544f5b45b6]
/lib/libc.so.6(cfree+0x73)[0x7f544f5bae83]
/usr/local/lib/libopenbabel.so.3(_ZN9OpenBabel15OBSmartsPattern5MatchERNS_5OBMolEb+0x3e)[0x7f5447c2b50e]
/usr/local/lib/libopenbabel.so.3(_ZN9OpenBabel11OBAtomTyper9AssignHybERNS_5OBMolE+0x108)[0x7f5447c59e88]
/usr/local/lib/libopenbabel.so.3(_ZNK9OpenBabel6OBAtom6GetHybEv+0x1f)[0x7f5447b9bd0f]
/usr/local/lib/libopenbabel.so.3(+0xce1dd)[0x7f5447c2a1dd]
/usr/local/lib/libopenbabel.so.3(_ZN9OpenBabel9OBSSMatch5MatchERSt6vectorIS1_IiSaIiEESaIS3_EEi+0x247)[0x7f5447c2a817]
/usr/local/lib/libopenbabel.so.3(+0xce8ff)[0x7f5447c2a8ff]
/usr/local/lib/libopenbabel.so.3(_ZN9OpenBabel15OBSmartsPattern5MatchERNS_5OBMolEb+0xd0)[0x7f5447c2b5a0]
/usr/local/lib/libopenbabel.so.3(_ZN9OpenBabel11OBAtomTyper21AssignImplicitValenceERNS_5OBMolE+0x130)[0x7f5447c595f0]
/usr/local/lib/libopenbabel.so.3(_ZNK9OpenBabel6OBAtom21ImplicitHydrogenCountEv+0x6c)[0x7f5447b9b9ec]
/usr/local/lib/libopenbabel.so.3(_ZN9OpenBabel5OBMol8GetMolWtEb+0x99)[0x7f5447bf8de9]
/usr/lib/mysql/plugin/libmychem.so(getMolWeight+0x229)[0x7f544a21f809]
/usr/lib/mysql/plugin/libmychem.so(molweight+0x67)[0x7f544a21d277]
/usr/sbin/mysqld(_ZN19Item_func_udf_float8val_realEv+0x4e)[0x7f54511f034e]
/usr/sbin/mysqld(_ZN12Item_func_if8val_realEv+0x3a)[0x7f545120384a]
/usr/sbin/mysqld(_ZN4Item13save_in_fieldEP5Fieldb+0x1c4)[0x7f54511c7804]
/usr/sbin/mysqld(+0x3ca782)[0x7f54512f0782]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x194)[0x7f54512f0fc4]
======= Memory map: ========
--
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
110908 6:26:32 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=4
max_threads=151
threads_connected=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346495 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x7f5440011560
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f54473f8e58 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7f54515781a9]
/usr/sbin/mysqld(handle_segfault+0x404) [0x7f5451285884]
/lib/libpthread.so.0(+0xf8f0) [0x7f5450af68f0]
/lib/libc.so.6(gsignal+0x35) [0x7f544f570a75]
/lib/libc.so.6(abort+0x180) [0x7f544f5745c0]
/lib/libc.so.6(+0x6d4fb) [0x7f544f5aa4fb]
/lib/libc.so.6(+0x775b6) [0x7f544f5b45b6]
/lib/libc.so.6(cfree+0x73) [0x7f544f5bae83]
/usr/local/lib/libopenbabel.so.3(OpenBabel::OBSmartsPattern::Match(OpenBabel::OBMol&, bool)+0x3e) [0x7f5447c2b50e]
/usr/local/lib/libopenbabel.so.3(OpenBabel::OBAtomTyper::AssignHyb(OpenBabel::OBMol&)+0x108) [0x7f5447c59e88]
/usr/local/lib/libopenbabel.so.3(OpenBabel::OBAtom::GetHyb() const+0x1f) [0x7f5447b9bd0f]
/usr/local/lib/libopenbabel.so.3(+0xce1dd) [0x7f5447c2a1dd]
/usr/local/lib/libopenbabel.so.3(OpenBabel::OBSSMatch::Match(std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >&, int)+0x247) [0x7f5447c2a817]
/usr/local/lib/libopenbabel.so.3(+0xce8ff) [0x7f5447c2a8ff]
/usr/local/lib/libopenbabel.so.3(OpenBabel::OBSmartsPattern::Match(OpenBabel::OBMol&, bool)+0xd0) [0x7f5447c2b5a0]
/usr/local/lib/libopenbabel.so.3(OpenBabel::OBAtomTyper::AssignImplicitValence(OpenBabel::OBMol&)+0x130) [0x7f5447c595f0]
/usr/local/lib/libopenbabel.so.3(OpenBabel::OBAtom::ImplicitHydrogenCount() const+0x6c) [0x7f5447b9b9ec]
/usr/local/lib/libopenbabel.so.3(OpenBabel::OBMol::GetMolWt(bool)+0x99) [0x7f5447bf8de9]
/usr/lib/mysql/plugin/libmychem.so(getMolWeight+0x229) [0x7f544a21f809]
/usr/lib/mysql/plugin/libmychem.so(molweight+0x67) [0x7f544a21d277]
/usr/sbin/mysqld(Item_func_udf_float::val_real()+0x4e) [0x7f54511f034e]
/usr/sbin/mysqld(Item_func_if::val_real()+0x3a) [0x7f545120384a]
/usr/sbin/mysqld(Item::save_in_field(Field*, bool)+0x1c4) [0x7f54511c7804]
/usr/sbin/mysqld(+0x3ca782) [0x7f54512f0782]
/usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x194) [0x7f54512f0fc4]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...

[Jerome Pansanel]

Hi Fredrik,

I think that this bug is due to multi threading. I will try to reproduce it on my server and check how to to solve this problem.

did you observe the problem with all functions, or only a subset?

Cheers,

Jerome

[Dr. Tyrell]

This should be fixed by turning off query_cache_strip_comments
see https://bugs.launchpad.net/percona-server/+bug/705688

[Jerome Pansanel]

It's a duplicate from the bug 3333590:
https://sourceforge.net/tracker/?func=detail&aid=3333590&group_id=195099&atid=952081

[Fredrik]

Hello,

I'm sorry, but I don't understand the solution offered by doctortyrell. As far as I understand, MySQL doesn't have the query_cache_strip_comments option, it seems to be specific to Percona. At least I get the response "ERROR 1193 (HY000): Unknown system variable 'query_cache_strip_comments'" when I tried to set it to 0 in my MySQL version (5.1.41-3ubuntu12.10)

[Fredrik]

Oh, I realized that I had forgotten to answer Jerome's question from september...
I see the problem at least in MOLWEIGHT() and MOLFORMULA(). The crash always happens in OBSmartsPattern::Match