Menu
▾
▴
Re: [libdbi-users] Found and fixed: dbd_mysql thread unsafe problem
|
From: Markus H. <mar...@mh...> - 2019-11-18 23:34:21
|
Hi Ethan and anyone else who might be listening, I'm sorry for the sluggish response time. I'm afraid I'm the one you're looking for. I used to maintain libdbi and the drivers for a couple of years while I was able to do some programming. I continued as sort of a caretaker for another while even though there was no time left for pretty much anything outside my dayjob. I still use libdbi everyday but I'd have to read the manual how to check in a patch these days. So long story short - if anyone is willing to take over development, or at least to act as a caretaker, please step forward. If you have the skills to check in and test your patch, let me know. I can add you as a developer to the project. regards, Markus On 2019-11-16 21:09, Ethan Funk was heard to say: > I found and tested a fix for the mysql thread safety problem I have > been troubleshooting. > > Commenting out line 154 of dbd_mysql.c succeeds in fixing the problem > when libdbi-mysql is build against recent libmysqlclient libraries. > This is the were mysql_server_end() is called after each dbi instance > if finished. In a multi-threaded application, multiple dbi instances > can come and go in various threads. With recent libmysqlclient library > implementations, mysql_server_end() should only be called ONCE, at the > end of the entire program/process, not multiple times when a thread is > finished with a dbi intsnace. > > libmysqlclient's mysql_server_end() function performs some final > libmysqlclient memory clean-up. So if it isn't called, and the > application ends, it's not really a problem - the OS will clean up the > memory allocation. > > A few other thread safety notes that might be helpful to others. To be > thread safe: > > 1. Each program thread must use it's own libdbi instance (as > documented in libdbi). > > 2. The dbi_conn_connect function must be protected by a mutex, since > it results in an underlying libmysqlclient call that is not thread > safe, using memory objects allocated once by the libmysqlclient > library when it was first initialized. This is exactly the memory that > was causing trouble (freed) when mysql_server_end() is called by > libdbi_shutdown_r in onw thread when the libdbi library is still in > use by other threads prior to program termination. > > So... is libdbi and the database drivers still maintained by someone? > How can I get this fix included so it will be present when folks > install the driver for mysql on their various systems? > > Thanks, > Ethan... > > On Fri, 2019-11-15 at 09:35 -0700, Ethan Funk wrote: > >> I think I found the problem in the mysql dbd driver code.: >> >> dbd_finalize() calls mysql_server_end(), which is deprecated in the >> >> current mysqlclient21 library, but results in a call to a >> replacement >> >> function which is intended to be called only once, at the end of a >> >> process. With the call being made inside dbd_finalize(), the >> >> mysql_server_end() gets called each time my application code >> finished >> >> with a libdbi instance. Reading between the lines in the MySQL >> >> documentation, it looks think the mysql_server_end() call to it's >> new >> >> counter part frees memory which is allocated by a pthread_once() >> >> function triggered by mysql_library_init(), which itself is called >> by >> >> mysql_init() via the libdbi driver on each connection attempt. So >> my >> >> code's first libdbi connection results in the memory being >> allocated, >> >> with subsequent calls in other threads being ignored. But >> >> dbd_finalize() gets called at the end of EVERY libdbi instance >> closure, >> >> so the memory if freed over and over again. I don't see a mechanism >> >> for a once-per-process driver shutdown callback in libdbi, so I want >> to >> >> try commenting out mysql_servefr_end() in dbd_finalize(). This will >> >> leak memory which is never freed at the end of the process, but so >> >> what. It is only allocated once per process, so the operating >> system >> >> process shutdown will free the memory. >> >> Unfortunately, I can't seem to build my modified code for testing >> with >> >> the libdi_dbd project code. "./configure" is giving me the following >> >> error: >> >> configure: error: Invalid libdbi directory - include files not >> found. >> >> This is after configure has already indicated that yes, it has >> located >> >> the dbi header files, which I installed with the libdbi-dev package >> >> with the Ubuntu apt command. >> >> How might I be able to fix this error, and proceed with testing my >> code >> >> fix? >> >> Thanks, >> >> Ethan... >> >> On Wed, 2019-11-13 at 14:19 -0700, Ethan Funk wrote: >> >>> I am having a problem with libdbd_mysql on linux (Ubuntu 19.10) >>> >>> reagding thread fasety. >>> >>> There used to be two versions of libmysqlclient, one was thread >>> safe, >>> >>> one was not. However, >>> >>> now there seems to be only one client library, which I understood >>> was >>> >>> thread safe: libmysqlclient.so.21. >>> >>> Any insight into the dump below would be greatly appreciated! >>> >>> Ethan... >>> >>> Here is a valgrind dump of a read-already-freed error that occurs >>> in >>> >>> what I will call thread #3 >>> >>> when it attempts to connect to MySQL. This thread has it's own >>> libdbi >>> >>> instance, yet mysql_real_connect >>> >>> is trying to read memory allocated by the first thread (thread #1) >>> that >>> >>> first called mysql_real_connect, >>> >>> after that same memory was freed by another thread (thread #2) >>> that >>> >>> called mysql_server_end as it was >>> >>> finishing. Each thread has it's own libdbi instance, so why is >>> >>> libmysqlclient sharring memory between >>> >>> threads? It's looking like the libmysqlclient isn't thread safe. >>> >>> ==2828== Invalid read of size 1 >>> >>> ==2828== at 0x4A3A042: strlen (vg_replace_strmem.c:461) >>> >>> ==2828== by 0x141A46C1: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A5468: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A54A8: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A373E: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x4FE041E: __pthread_once_slow (pthread_once.c:116) >>> >>> ==2828== by 0x141A320D: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A41B2: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A429E: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x1415472B: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x1415492A: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64- >>> >>> linux-gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux- >>> >>> gnu/dbd/libdbdmysql.so) >>> >>> ==2828== by 0x4DC37AE: dbi_conn_connect (in >>> /usr/lib/x86_64-linux- >>> >>> gnu/libdbi.so.1.1.0) >>> >>> ==2828== by 0x134BFA: dbSetupConnection (database.c:211) >>> >>> ==2828== by 0x1357F6: DeleteLogEntry (database.c:440) >>> >>> ==2828== by 0x131ECE: taskCallProcInThread (tasks.c:73) >>> >>> ==2828== by 0x4FD7668: start_thread (pthread_create.c:479) >>> >>> ==2828== by 0x567B322: clone (clone.S:95) >>> >>> Memory was freed in a thread (lets call it thread #2) that ran and >>> >>> completed. It created it's own >>> >>> dbi instance, did it's thing, and then closed it's MySQL >>> connection and >>> >>> shutdown the instance. >>> >>> So why is it freeing memory deep in the mysql driver that it did >>> not >>> >>> allocate? >>> >>> ==2828== Address 0x1343e698 is 424 bytes inside a block of size >>> 4,088 >>> >>> free'd >>> >>> ==2828== at 0x4A37F90: free (vg_replace_malloc.c:540) >>> >>> ==2828== by 0x141A9B8A: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A8756: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x14148982: mysql_server_end (in >>> /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x12F4684C: dbd_finalize (in /usr/lib/x86_64-linux- >>> >>> gnu/dbd/libdbdmysql.so) >>> >>> ==2828== by 0x4DC2E4D: dbi_shutdown_r (in >>> /usr/lib/x86_64-linux- >>> >>> gnu/libdbi.so.1.1.0) >>> >>> ==2828== by 0x134440: dbi_instance_free (database.c:56) >>> >>> ==2828== by 0x4FD6600: __nptl_deallocate_tsd.part.0 >>> >>> (pthread_create.c:301) >>> >>> ==2828== by 0x4FD7689: __nptl_deallocate_tsd >>> (pthread_create.c:256) >>> >>> ==2828== by 0x4FD7689: start_thread (pthread_create.c:490) >>> >>> ==2828== by 0x567B322: clone (clone.S:95) >>> >>> This is in the first thread in the program to make use of >>> libdbi/MySQL, >>> >>> and it is still >>> >>> running/connected to MySQL using a unique libdbi intance. >>> >>> ==2828== Block was alloc'd at >>> >>> ==2828== at 0x4A36E96: malloc (vg_replace_malloc.c:309) >>> >>> ==2828== by 0x141A9A30: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A9AEB: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A4E1C: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x1419CFA0: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x1419D95E: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x1416FD29: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A2A66: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A3771: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x4FE041E: __pthread_once_slow (pthread_once.c:116) >>> >>> ==2828== by 0x141A320D: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A41B2: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141A429E: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x1415472B: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x1415492A: ??? (in /usr/lib/x86_64-linux- >>> >>> gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64- >>> >>> linux-gnu/libmysqlclient.so.21.1.17) >>> >>> ==2828== by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux- >>> >>> gnu/dbd/libdbdmysql.so) >>> >>> ==2828== by 0x4DC37AE: dbi_conn_connect (in >>> /usr/lib/x86_64-linux- >>> >>> gnu/libdbi.so.1.1.0) >>> >>> ==2828== by 0x134BFA: dbSetupConnection (database.c:211) >>> >>> ==2828== by 0x13D2F4: GetItemMetaData (database.c:2960) >>> >>> ==2828== > _______________________________________________ > libdbi-users mailing list > lib...@li... > https://lists.sourceforge.net/lists/listinfo/libdbi-users -- Markus Hoenicka http://www.mhoenicka.de AQ score 38 |
