libdbi-users — General libdbi discussion and support

Re: [libdbi-users] Found and fixed: dbd_mysql thread unsafe problem

[Ethan F. <et...@re...>]

Markus,
Unfortunately, I have no experience maintaining code for packages.  The
best I can do is a few lines in a Makefile, and I don't understand
automake scripts at all. I can fix all sorts of things in c source
code, but I would likely break the test and build process. So I
wouldn't trust me as a maintainer.
Thanks for your work on libdbi over the years.  I have been using it
for well over a decade.
Ethan...
On Tue, 2019-11-19 at 00:20 +0100, Markus Hoenicka wrote:
> Hi Ethan and anyone else who might be listening,
> I'm sorry for the sluggish response time. I'm afraid I'm the one
> you're looking for. I used to maintain libdbi and the drivers for a
> couple of years while I was able to do some programming. I continued
> as sort of a caretaker for another while even though there was no
> time left for pretty much anything outside my dayjob. I still use
> libdbi everyday but I'd have to read the manual how to check in a
> patch these days. So long story short - if anyone is willing to take
> over development, or at least to act as a caretaker, please step
> forward. If you have the skills to check in and test your patch, let
> me know. I can add you as a developer to the project.
> regards,Markus
> On 2019-11-16 21:09, Ethan Funk was heard to say:
> > I found and tested a fix for the mysql thread safety problem I
> > havebeen troubleshooting.
> > Commenting out line 154 of dbd_mysql.c succeeds in fixing the
> > problemwhen libdbi-mysql is build against recent libmysqlclient
> > libraries.This is the were mysql_server_end() is called after each
> > dbi instanceif finished. In a multi-threaded application, multiple
> > dbi instancescan come and go in various threads. With recent
> > libmysqlclient libraryimplementations, mysql_server_end() should
> > only be called ONCE, at theend of the entire program/process, not
> > multiple times when a thread isfinished with a dbi intsnace.
> > libmysqlclient's mysql_server_end() function performs some
> > finallibmysqlclient memory clean-up. So if it isn't called, and
> > theapplication ends, it's not really a problem - the OS will clean
> > up thememory allocation.
> > A few other thread safety notes that might be helpful to others. To
> > bethread safe:
> > 1. Each program thread must use it's own libdbi instance
> > (asdocumented in libdbi).
> > 2. The dbi_conn_connect function must be protected by a mutex,
> > sinceit results in an underlying libmysqlclient call that is not
> > threadsafe, using memory objects allocated once by the
> > libmysqlclientlibrary when it was first initialized. This is
> > exactly the memory thatwas causing trouble (freed) when
> > mysql_server_end() is called bylibdbi_shutdown_r in onw thread when
> > the libdbi library is still inuse by other threads prior to program
> > termination.
> > So... is libdbi and the database drivers still maintained by
> > someone?How can I get this fix included so it will be present when
> > folksinstall the driver for mysql on their various systems?
> > Thanks,Ethan...
> > On Fri, 2019-11-15 at 09:35 -0700, Ethan Funk wrote:
> > > I think I found the problem in the mysql dbd driver code.:
> > > dbd_finalize() calls mysql_server_end(), which is deprecated in
> > > the
> > > current mysqlclient21 library, but results in a call to
> > > areplacement
> > > function which is intended to be called only once, at the end of
> > > a
> > > process.  With the call being made inside dbd_finalize(), the
> > > mysql_server_end() gets called each time my application
> > > codefinished
> > > with a libdbi instance.  Reading between the lines in the MySQL
> > > documentation, it looks think the mysql_server_end() call to
> > > it'snew
> > > counter part frees memory which is allocated by a pthread_once()
> > > function triggered by mysql_library_init(), which itself is
> > > calledby
> > > mysql_init() via the libdbi driver on each connection
> > > attempt.  Somy
> > > code's first libdbi connection results in the memory
> > > beingallocated,
> > > with subsequent calls in other threads being ignored.  But
> > > dbd_finalize() gets called at the end of EVERY libdbi
> > > instanceclosure,
> > > so the memory if freed over and over again.  I don't see a
> > > mechanism
> > > for a once-per-process driver shutdown callback in libdbi, so I
> > > wantto
> > > try commenting out mysql_servefr_end() in dbd_finalize().  This
> > > will
> > > leak memory which is never freed at the end of the process, but
> > > so
> > > what.  It is only allocated once per process, so the
> > > operatingsystem
> > > process shutdown will free the memory.
> > > Unfortunately, I can't seem to build my modified code for
> > > testingwith
> > > the libdi_dbd project code. "./configure" is giving me the
> > > following
> > > error:
> > > configure: error: Invalid libdbi directory - include files
> > > notfound.
> > > This is after configure has already indicated that yes, it
> > > haslocated
> > > the dbi header files, which I installed with the libdbi-dev
> > > package
> > > with the Ubuntu apt command.
> > > How might I be able to fix this error, and proceed with testing
> > > mycode
> > > fix?
> > > Thanks,
> > > Ethan...
> > > On Wed, 2019-11-13 at 14:19 -0700, Ethan Funk wrote:
> > > > I am having a problem with libdbd_mysql on linux (Ubuntu 19.10)
> > > > reagding thread fasety.
> > > > There used to be two versions of libmysqlclient, one was
> > > > threadsafe,
> > > > one was not. However,
> > > > now there seems to be only one client library, which I
> > > > understoodwas
> > > > thread safe: libmysqlclient.so.21.
> > > > Any insight into the dump below would be greatly appreciated!
> > > > Ethan...
> > > > Here is a valgrind dump of a read-already-freed error that
> > > > occursin
> > > > what I will call thread #3
> > > > when it attempts to connect to MySQL.  This thread has it's
> > > > ownlibdbi
> > > > instance, yet mysql_real_connect
> > > > is trying to read memory allocated by the first thread (thread
> > > > #1)that
> > > > first called mysql_real_connect,
> > > > after that same memory was freed by another thread (thread
> > > > #2)that
> > > > called mysql_server_end as it was
> > > > finishing. Each thread has it's own libdbi instance, so why is
> > > > libmysqlclient sharring memory between
> > > > threads? It's looking like the libmysqlclient isn't thread
> > > > safe.
> > > > ==2828== Invalid read of size 1
> > > > ==2828==    at 0x4A3A042: strlen (vg_replace_strmem.c:461)
> > > > ==2828==    by 0x141A46C1: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A5468: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A54A8: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A373E: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x4FE041E: __pthread_once_slow
> > > > (pthread_once.c:116)
> > > > ==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141577A8: mysql_real_connect (in
> > > > /usr/lib/x86_64-
> > > > linux-gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-
> > > > linux-
> > > > gnu/dbd/libdbdmysql.so)
> > > > ==2828==    by 0x4DC37AE: dbi_conn_connect (in/usr/lib/x86_64-
> > > > linux-
> > > > gnu/libdbi.so.1.1.0)
> > > > ==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
> > > > ==2828==    by 0x1357F6: DeleteLogEntry (database.c:440)
> > > > ==2828==    by 0x131ECE: taskCallProcInThread (tasks.c:73)
> > > > ==2828==    by 0x4FD7668: start_thread (pthread_create.c:479)
> > > > ==2828==    by 0x567B322: clone (clone.S:95)
> > > > Memory was freed in a thread (lets call it thread #2) that ran
> > > > and
> > > > completed. It created it's own
> > > > dbi instance, did it's thing, and then closed it's
> > > > MySQLconnection and
> > > > shutdown the instance.
> > > > So why is it freeing memory deep in the mysql driver that it
> > > > didnot
> > > > allocate?
> > > > ==2828==  Address 0x1343e698 is 424 bytes inside a block of
> > > > size4,088
> > > > free'd
> > > > ==2828==    at 0x4A37F90: free (vg_replace_malloc.c:540)
> > > > ==2828==    by 0x141A9B8A: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A8756: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x14148982: mysql_server_end (in/usr/lib/x86_64-
> > > > linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x12F4684C: dbd_finalize (in /usr/lib/x86_64-
> > > > linux-
> > > > gnu/dbd/libdbdmysql.so)
> > > > ==2828==    by 0x4DC2E4D: dbi_shutdown_r (in/usr/lib/x86_64-
> > > > linux-
> > > > gnu/libdbi.so.1.1.0)
> > > > ==2828==    by 0x134440: dbi_instance_free (database.c:56)
> > > > ==2828==    by 0x4FD6600: __nptl_deallocate_tsd.part.0
> > > > (pthread_create.c:301)
> > > > ==2828==    by 0x4FD7689:
> > > > __nptl_deallocate_tsd(pthread_create.c:256)
> > > > ==2828==    by 0x4FD7689: start_thread (pthread_create.c:490)
> > > > ==2828==    by 0x567B322: clone (clone.S:95)
> > > > This is in the first thread in the program to make use
> > > > oflibdbi/MySQL,
> > > > and it is still
> > > > running/connected to MySQL using a unique libdbi intance.
> > > > ==2828==  Block was alloc'd at
> > > > ==2828==    at 0x4A36E96: malloc (vg_replace_malloc.c:309)
> > > > ==2828==    by 0x141A9A30: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A9AEB: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A4E1C: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x1419CFA0: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x1419D95E: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x1416FD29: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A2A66: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A3771: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x4FE041E: __pthread_once_slow
> > > > (pthread_once.c:116)
> > > > ==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
> > > > gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x141577A8: mysql_real_connect (in
> > > > /usr/lib/x86_64-
> > > > linux-gnu/libmysqlclient.so.21.1.17)
> > > > ==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-
> > > > linux-
> > > > gnu/dbd/libdbdmysql.so)
> > > > ==2828==    by 0x4DC37AE: dbi_conn_connect (in/usr/lib/x86_64-
> > > > linux-
> > > > gnu/libdbi.so.1.1.0)
> > > > ==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
> > > > ==2828==    by 0x13D2F4: GetItemMetaData (database.c:2960)
> > > > ==2828==
> > _______________________________________________libdbi-users mailing
> > lis...@li...
> > https://lists.sourceforge.net/lists/listinfo/libdbi-users

Re: [libdbi-users] Found and fixed: dbd_mysql thread unsafe problem

[Markus H. <mar...@mh...>]

Hi Ethan and anyone else who might be listening,

I'm sorry for the sluggish response time. I'm afraid I'm the one you're 
looking for. I used to maintain libdbi and the drivers for a couple of 
years while I was able to do some programming. I continued as sort of a 
caretaker for another while even though there was no time left for 
pretty much anything outside my dayjob. I still use libdbi everyday but 
I'd have to read the manual how to check in a patch these days. So long 
story short - if anyone is willing to take over development, or at least 
to act as a caretaker, please step forward. If you have the skills to 
check in and test your patch, let me know. I can add you as a developer 
to the project.

regards,
Markus

On 2019-11-16 21:09, Ethan Funk was heard to say:
> I found and tested a fix for the mysql thread safety problem I have
> been troubleshooting.
> 
> Commenting out line 154 of dbd_mysql.c succeeds in fixing the problem
> when libdbi-mysql is build against recent libmysqlclient libraries.
> This is the were mysql_server_end() is called after each dbi instance
> if finished. In a multi-threaded application, multiple dbi instances
> can come and go in various threads. With recent libmysqlclient library
> implementations, mysql_server_end() should only be called ONCE, at the
> end of the entire program/process, not multiple times when a thread is
> finished with a dbi intsnace.
> 
> libmysqlclient's mysql_server_end() function performs some final
> libmysqlclient memory clean-up. So if it isn't called, and the
> application ends, it's not really a problem - the OS will clean up the
> memory allocation.
> 
> A few other thread safety notes that might be helpful to others. To be
> thread safe:
> 
> 1. Each program thread must use it's own libdbi instance (as
> documented in libdbi).
> 
> 2. The dbi_conn_connect function must be protected by a mutex, since
> it results in an underlying libmysqlclient call that is not thread
> safe, using memory objects allocated once by the libmysqlclient
> library when it was first initialized. This is exactly the memory that
> was causing trouble (freed) when mysql_server_end() is called by
> libdbi_shutdown_r in onw thread when the libdbi library is still in
> use by other threads prior to program termination.
> 
> So... is libdbi and the database drivers still maintained by someone?
> How can I get this fix included so it will be present when folks
> install the driver for mysql on their various systems?
> 
> Thanks,
> Ethan...
> 
> On Fri, 2019-11-15 at 09:35 -0700, Ethan Funk wrote:
> 
>> I think I found the problem in the mysql dbd driver code.:
>> 
>> dbd_finalize() calls mysql_server_end(), which is deprecated in the
>> 
>> current mysqlclient21 library, but results in a call to a
>> replacement
>> 
>> function which is intended to be called only once, at the end of a
>> 
>> process.  With the call being made inside dbd_finalize(), the
>> 
>> mysql_server_end() gets called each time my application code
>> finished
>> 
>> with a libdbi instance.  Reading between the lines in the MySQL
>> 
>> documentation, it looks think the mysql_server_end() call to it's
>> new
>> 
>> counter part frees memory which is allocated by a pthread_once()
>> 
>> function triggered by mysql_library_init(), which itself is called
>> by
>> 
>> mysql_init() via the libdbi driver on each connection attempt.  So
>> my
>> 
>> code's first libdbi connection results in the memory being
>> allocated,
>> 
>> with subsequent calls in other threads being ignored.  But
>> 
>> dbd_finalize() gets called at the end of EVERY libdbi instance
>> closure,
>> 
>> so the memory if freed over and over again.  I don't see a mechanism
>> 
>> for a once-per-process driver shutdown callback in libdbi, so I want
>> to
>> 
>> try commenting out mysql_servefr_end() in dbd_finalize().  This will
>> 
>> leak memory which is never freed at the end of the process, but so
>> 
>> what.  It is only allocated once per process, so the operating
>> system
>> 
>> process shutdown will free the memory.
>> 
>> Unfortunately, I can't seem to build my modified code for testing
>> with
>> 
>> the libdi_dbd project code. "./configure" is giving me the following
>> 
>> error:
>> 
>> configure: error: Invalid libdbi directory - include files not
>> found.
>> 
>> This is after configure has already indicated that yes, it has
>> located
>> 
>> the dbi header files, which I installed with the libdbi-dev package
>> 
>> with the Ubuntu apt command.
>> 
>> How might I be able to fix this error, and proceed with testing my
>> code
>> 
>> fix?
>> 
>> Thanks,
>> 
>> Ethan...
>> 
>> On Wed, 2019-11-13 at 14:19 -0700, Ethan Funk wrote:
>> 
>>> I am having a problem with libdbd_mysql on linux (Ubuntu 19.10)
>>> 
>>> reagding thread fasety.
>>> 
>>> There used to be two versions of libmysqlclient, one was thread
>>> safe,
>>> 
>>> one was not. However,
>>> 
>>> now there seems to be only one client library, which I understood
>>> was
>>> 
>>> thread safe: libmysqlclient.so.21.
>>> 
>>> Any insight into the dump below would be greatly appreciated!
>>> 
>>> Ethan...
>>> 
>>> Here is a valgrind dump of a read-already-freed error that occurs
>>> in
>>> 
>>> what I will call thread #3
>>> 
>>> when it attempts to connect to MySQL.  This thread has it's own
>>> libdbi
>>> 
>>> instance, yet mysql_real_connect
>>> 
>>> is trying to read memory allocated by the first thread (thread #1)
>>> that
>>> 
>>> first called mysql_real_connect,
>>> 
>>> after that same memory was freed by another thread (thread #2)
>>> that
>>> 
>>> called mysql_server_end as it was
>>> 
>>> finishing. Each thread has it's own libdbi instance, so why is
>>> 
>>> libmysqlclient sharring memory between
>>> 
>>> threads? It's looking like the libmysqlclient isn't thread safe.
>>> 
>>> ==2828== Invalid read of size 1
>>> 
>>> ==2828==    at 0x4A3A042: strlen (vg_replace_strmem.c:461)
>>> 
>>> ==2828==    by 0x141A46C1: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A5468: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A54A8: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A373E: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x4FE041E: __pthread_once_slow (pthread_once.c:116)
>>> 
>>> ==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64-
>>> 
>>> linux-gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/dbd/libdbdmysql.so)
>>> 
>>> ==2828==    by 0x4DC37AE: dbi_conn_connect (in
>>> /usr/lib/x86_64-linux-
>>> 
>>> gnu/libdbi.so.1.1.0)
>>> 
>>> ==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
>>> 
>>> ==2828==    by 0x1357F6: DeleteLogEntry (database.c:440)
>>> 
>>> ==2828==    by 0x131ECE: taskCallProcInThread (tasks.c:73)
>>> 
>>> ==2828==    by 0x4FD7668: start_thread (pthread_create.c:479)
>>> 
>>> ==2828==    by 0x567B322: clone (clone.S:95)
>>> 
>>> Memory was freed in a thread (lets call it thread #2) that ran and
>>> 
>>> completed. It created it's own
>>> 
>>> dbi instance, did it's thing, and then closed it's MySQL
>>> connection and
>>> 
>>> shutdown the instance.
>>> 
>>> So why is it freeing memory deep in the mysql driver that it did
>>> not
>>> 
>>> allocate?
>>> 
>>> ==2828==  Address 0x1343e698 is 424 bytes inside a block of size
>>> 4,088
>>> 
>>> free'd
>>> 
>>> ==2828==    at 0x4A37F90: free (vg_replace_malloc.c:540)
>>> 
>>> ==2828==    by 0x141A9B8A: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A8756: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x14148982: mysql_server_end (in
>>> /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x12F4684C: dbd_finalize (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/dbd/libdbdmysql.so)
>>> 
>>> ==2828==    by 0x4DC2E4D: dbi_shutdown_r (in
>>> /usr/lib/x86_64-linux-
>>> 
>>> gnu/libdbi.so.1.1.0)
>>> 
>>> ==2828==    by 0x134440: dbi_instance_free (database.c:56)
>>> 
>>> ==2828==    by 0x4FD6600: __nptl_deallocate_tsd.part.0
>>> 
>>> (pthread_create.c:301)
>>> 
>>> ==2828==    by 0x4FD7689: __nptl_deallocate_tsd
>>> (pthread_create.c:256)
>>> 
>>> ==2828==    by 0x4FD7689: start_thread (pthread_create.c:490)
>>> 
>>> ==2828==    by 0x567B322: clone (clone.S:95)
>>> 
>>> This is in the first thread in the program to make use of
>>> libdbi/MySQL,
>>> 
>>> and it is still
>>> 
>>> running/connected to MySQL using a unique libdbi intance.
>>> 
>>> ==2828==  Block was alloc'd at
>>> 
>>> ==2828==    at 0x4A36E96: malloc (vg_replace_malloc.c:309)
>>> 
>>> ==2828==    by 0x141A9A30: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A9AEB: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A4E1C: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x1419CFA0: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x1419D95E: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x1416FD29: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A2A66: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A3771: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x4FE041E: __pthread_once_slow (pthread_once.c:116)
>>> 
>>> ==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64-
>>> 
>>> linux-gnu/libmysqlclient.so.21.1.17)
>>> 
>>> ==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux-
>>> 
>>> gnu/dbd/libdbdmysql.so)
>>> 
>>> ==2828==    by 0x4DC37AE: dbi_conn_connect (in
>>> /usr/lib/x86_64-linux-
>>> 
>>> gnu/libdbi.so.1.1.0)
>>> 
>>> ==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
>>> 
>>> ==2828==    by 0x13D2F4: GetItemMetaData (database.c:2960)
>>> 
>>> ==2828==
> _______________________________________________
> libdbi-users mailing list
> lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libdbi-users

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[libdbi-users] Found and fixed: dbd_mysql thread unsafe problem

[Ethan F. <et...@re...>]

I found and tested a fix for the mysql thread safety problem I have
been troubleshooting.

Commenting out line 154 of dbd_mysql.c succeeds in fixing the problem
when libdbi-mysql is build against recent libmysqlclient libraries.
This is the were mysql_server_end() is called after each dbi instance
if finished.  In a multi-threaded application, multiple dbi instances
can come and go in various threads.  With recent libmysqlclient library
implementations, mysql_server_end() should only be called ONCE, at the
end of the entire program/process, not multiple times when a thread is
finished with a dbi intsnace.

libmysqlclient's mysql_server_end() function performs some final
libmysqlclient memory clean-up.  So if it isn't called, and the
application ends, it's not really a problem -  the OS will clean up the
memory allocation.

A few other thread safety notes that might be helpful to others. To be
thread safe:

1. Each program thread must use it's own libdbi instance (as documented
in libdbi).

2. The dbi_conn_connect function must be protected by a mutex, since it
results in an underlying libmysqlclient call that is not thread safe,
using memory objects allocated once by the libmysqlclient library when
it was first initialized.  This is exactly the memory that was causing
trouble (freed) when mysql_server_end() is called by libdbi_shutdown_r
in onw thread when the libdbi library is still in use by other threads
prior to program termination.

So... is libdbi and the database drivers still maintained by someone? 
How can I get this fix included so it will be present when folks
install the driver for mysql on their various systems?

Thanks,
Ethan...

On Fri, 2019-11-15 at 09:35 -0700, Ethan Funk wrote:
> I think I found the problem in the mysql dbd driver code.:
> 
> dbd_finalize() calls mysql_server_end(), which is deprecated in the
> current mysqlclient21 library, but results in a call to a replacement
> function which is intended to be called only once, at the end of a
> process.  With the call being made inside dbd_finalize(), the
> mysql_server_end() gets called each time my application code finished
> with a libdbi instance.  Reading between the lines in the MySQL
> documentation, it looks think the mysql_server_end() call to it's new
> counter part frees memory which is allocated by a pthread_once()
> function triggered by mysql_library_init(), which itself is called by
> mysql_init() via the libdbi driver on each connection attempt.  So my
> code's first libdbi connection results in the memory being allocated,
> with subsequent calls in other threads being ignored.  But
> dbd_finalize() gets called at the end of EVERY libdbi instance closure,
> so the memory if freed over and over again.  I don't see a mechanism
> for a once-per-process driver shutdown callback in libdbi, so I want to
> try commenting out mysql_servefr_end() in dbd_finalize().  This will
> leak memory which is never freed at the end of the process, but so
> what.  It is only allocated once per process, so the operating system
> process shutdown will free the memory.
> 
> Unfortunately, I can't seem to build my modified code for testing with
> the libdi_dbd project code. "./configure" is giving me the following
> error:
> configure: error: Invalid libdbi directory - include files not found.
> 
> This is after configure has already indicated that yes, it has located
> the dbi header files, which I installed with the libdbi-dev package
> with the Ubuntu apt command.
> 
> How might I be able to fix this error, and proceed with testing my code
> fix?
> 
> Thanks,
> Ethan...
> 
> On Wed, 2019-11-13 at 14:19 -0700, Ethan Funk wrote:
> > I am having a problem with libdbd_mysql on linux (Ubuntu 19.10)
> > reagding thread fasety.
> > There used to be two versions of libmysqlclient, one was thread safe,
> > one was not. However,
> > now there seems to be only one client library, which I understood was
> > thread safe: libmysqlclient.so.21.
> > 
> > Any insight into the dump below would be greatly appreciated!
> > 
> > Ethan...
> > 
> > Here is a valgrind dump of a read-already-freed error that occurs in
> > what I will call thread #3
> > when it attempts to connect to MySQL.  This thread has it's own libdbi
> > instance, yet mysql_real_connect
> > is trying to read memory allocated by the first thread (thread #1) that
> > first called mysql_real_connect, 
> > after that same memory was freed by another thread (thread #2) that
> > called mysql_server_end as it was 
> > finishing. Each thread has it's own libdbi instance, so why is
> > libmysqlclient sharring memory between 
> > threads? It's looking like the libmysqlclient isn't thread safe.
> > 
> > ==2828== Invalid read of size 1
> > ==2828==    at 0x4A3A042: strlen (vg_replace_strmem.c:461)
> > ==2828==    by 0x141A46C1: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A5468: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A54A8: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A373E: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x4FE041E: __pthread_once_slow (pthread_once.c:116)
> > ==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64-
> > linux-gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux-
> > gnu/dbd/libdbdmysql.so)
> > ==2828==    by 0x4DC37AE: dbi_conn_connect (in /usr/lib/x86_64-linux-
> > gnu/libdbi.so.1.1.0)
> > ==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
> > ==2828==    by 0x1357F6: DeleteLogEntry (database.c:440)
> > ==2828==    by 0x131ECE: taskCallProcInThread (tasks.c:73)
> > ==2828==    by 0x4FD7668: start_thread (pthread_create.c:479)
> > ==2828==    by 0x567B322: clone (clone.S:95)
> > 
> > Memory was freed in a thread (lets call it thread #2) that ran and
> > completed. It created it's own 
> > dbi instance, did it's thing, and then closed it's MySQL connection and
> > shutdown the instance.
> > So why is it freeing memory deep in the mysql driver that it did not
> > allocate?
> > 
> > ==2828==  Address 0x1343e698 is 424 bytes inside a block of size 4,088
> > free'd
> > ==2828==    at 0x4A37F90: free (vg_replace_malloc.c:540)
> > ==2828==    by 0x141A9B8A: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A8756: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x14148982: mysql_server_end (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x12F4684C: dbd_finalize (in /usr/lib/x86_64-linux-
> > gnu/dbd/libdbdmysql.so)
> > ==2828==    by 0x4DC2E4D: dbi_shutdown_r (in /usr/lib/x86_64-linux-
> > gnu/libdbi.so.1.1.0)
> > ==2828==    by 0x134440: dbi_instance_free (database.c:56)
> > ==2828==    by 0x4FD6600: __nptl_deallocate_tsd.part.0
> > (pthread_create.c:301)
> > ==2828==    by 0x4FD7689: __nptl_deallocate_tsd (pthread_create.c:256)
> > ==2828==    by 0x4FD7689: start_thread (pthread_create.c:490)
> > ==2828==    by 0x567B322: clone (clone.S:95)
> > 
> > This is in the first thread in the program to make use of libdbi/MySQL,
> > and it is still 
> > running/connected to MySQL using a unique libdbi intance.
> > 
> > ==2828==  Block was alloc'd at
> > ==2828==    at 0x4A36E96: malloc (vg_replace_malloc.c:309)
> > ==2828==    by 0x141A9A30: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A9AEB: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A4E1C: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x1419CFA0: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x1419D95E: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x1416FD29: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A2A66: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A3771: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x4FE041E: __pthread_once_slow (pthread_once.c:116)
> > ==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
> > gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64-
> > linux-gnu/libmysqlclient.so.21.1.17)
> > ==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux-
> > gnu/dbd/libdbdmysql.so)
> > ==2828==    by 0x4DC37AE: dbi_conn_connect (in /usr/lib/x86_64-linux-
> > gnu/libdbi.so.1.1.0)
> > ==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
> > ==2828==    by 0x13D2F4: GetItemMetaData (database.c:2960)
> > ==2828== 

Re: [libdbi-users] Possible dbd_mysql thread unsafe problem?

[Ethan F. <et...@re...>]

I think I found the problem in the mysql dbd driver code.:

dbd_finalize() calls mysql_server_end(), which is deprecated in the
current mysqlclient21 library, but results in a call to a replacement
function which is intended to be called only once, at the end of a
process.  With the call being made inside dbd_finalize(), the
mysql_server_end() gets called each time my application code finished
with a libdbi instance.  Reading between the lines in the MySQL
documentation, it looks think the mysql_server_end() call to it's new
counter part frees memory which is allocated by a pthread_once()
function triggered by mysql_library_init(), which itself is called by
mysql_init() via the libdbi driver on each connection attempt.  So my
code's first libdbi connection results in the memory being allocated,
with subsequent calls in other threads being ignored.  But
dbd_finalize() gets called at the end of EVERY libdbi instance closure,
so the memory if freed over and over again.  I don't see a mechanism
for a once-per-process driver shutdown callback in libdbi, so I want to
try commenting out mysql_servefr_end() in dbd_finalize().  This will
leak memory which is never freed at the end of the process, but so
what.  It is only allocated once per process, so the operating system
process shutdown will free the memory.

Unfortunately, I can't seem to build my modified code for testing with
the libdi_dbd project code. "./configure" is giving me the following
error:
configure: error: Invalid libdbi directory - include files not found.

This is after configure has already indicated that yes, it has located
the dbi header files, which I installed with the libdbi-dev package
with the Ubuntu apt command.

How might I be able to fix this error, and proceed with testing my code
fix?

Thanks,
Ethan...

On Wed, 2019-11-13 at 14:19 -0700, Ethan Funk wrote:
> I am having a problem with libdbd_mysql on linux (Ubuntu 19.10)
> reagding thread fasety.
> There used to be two versions of libmysqlclient, one was thread safe,
> one was not. However,
> now there seems to be only one client library, which I understood was
> thread safe: libmysqlclient.so.21.
> 
> Any insight into the dump below would be greatly appreciated!
> 
> Ethan...
> 
> Here is a valgrind dump of a read-already-freed error that occurs in
> what I will call thread #3
> when it attempts to connect to MySQL.  This thread has it's own libdbi
> instance, yet mysql_real_connect
> is trying to read memory allocated by the first thread (thread #1) that
> first called mysql_real_connect, 
> after that same memory was freed by another thread (thread #2) that
> called mysql_server_end as it was 
> finishing. Each thread has it's own libdbi instance, so why is
> libmysqlclient sharring memory between 
> threads? It's looking like the libmysqlclient isn't thread safe.
> 
> ==2828== Invalid read of size 1
> ==2828==    at 0x4A3A042: strlen (vg_replace_strmem.c:461)
> ==2828==    by 0x141A46C1: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A5468: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A54A8: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A373E: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x4FE041E: __pthread_once_slow (pthread_once.c:116)
> ==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64-
> linux-gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux-
> gnu/dbd/libdbdmysql.so)
> ==2828==    by 0x4DC37AE: dbi_conn_connect (in /usr/lib/x86_64-linux-
> gnu/libdbi.so.1.1.0)
> ==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
> ==2828==    by 0x1357F6: DeleteLogEntry (database.c:440)
> ==2828==    by 0x131ECE: taskCallProcInThread (tasks.c:73)
> ==2828==    by 0x4FD7668: start_thread (pthread_create.c:479)
> ==2828==    by 0x567B322: clone (clone.S:95)
> 
> Memory was freed in a thread (lets call it thread #2) that ran and
> completed. It created it's own 
> dbi instance, did it's thing, and then closed it's MySQL connection and
> shutdown the instance.
> So why is it freeing memory deep in the mysql driver that it did not
> allocate?
> 
> ==2828==  Address 0x1343e698 is 424 bytes inside a block of size 4,088
> free'd
> ==2828==    at 0x4A37F90: free (vg_replace_malloc.c:540)
> ==2828==    by 0x141A9B8A: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A8756: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x14148982: mysql_server_end (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x12F4684C: dbd_finalize (in /usr/lib/x86_64-linux-
> gnu/dbd/libdbdmysql.so)
> ==2828==    by 0x4DC2E4D: dbi_shutdown_r (in /usr/lib/x86_64-linux-
> gnu/libdbi.so.1.1.0)
> ==2828==    by 0x134440: dbi_instance_free (database.c:56)
> ==2828==    by 0x4FD6600: __nptl_deallocate_tsd.part.0
> (pthread_create.c:301)
> ==2828==    by 0x4FD7689: __nptl_deallocate_tsd (pthread_create.c:256)
> ==2828==    by 0x4FD7689: start_thread (pthread_create.c:490)
> ==2828==    by 0x567B322: clone (clone.S:95)
> 
> This is in the first thread in the program to make use of libdbi/MySQL,
> and it is still 
> running/connected to MySQL using a unique libdbi intance.
> 
> ==2828==  Block was alloc'd at
> ==2828==    at 0x4A36E96: malloc (vg_replace_malloc.c:309)
> ==2828==    by 0x141A9A30: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A9AEB: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A4E1C: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x1419CFA0: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x1419D95E: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x1416FD29: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A2A66: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A3771: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x4FE041E: __pthread_once_slow (pthread_once.c:116)
> ==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
> gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64-
> linux-gnu/libmysqlclient.so.21.1.17)
> ==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux-
> gnu/dbd/libdbdmysql.so)
> ==2828==    by 0x4DC37AE: dbi_conn_connect (in /usr/lib/x86_64-linux-
> gnu/libdbi.so.1.1.0)
> ==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
> ==2828==    by 0x13D2F4: GetItemMetaData (database.c:2960)
> ==2828== 

[libdbi-users] Possible dbd_mysql thread unsafe problem?

[Ethan F. <et...@re...>]

I am having a problem with libdbd_mysql on linux (Ubuntu 19.10)
reagding thread fasety.
There used to be two versions of libmysqlclient, one was thread safe,
one was not. However,
now there seems to be only one client library, which I understood was
thread safe: libmysqlclient.so.21.

Any insight into the dump below would be greatly appreciated!

Ethan...

Here is a valgrind dump of a read-already-freed error that occurs in
what I will call thread #3
when it attempts to connect to MySQL.  This thread has it's own libdbi
instance, yet mysql_real_connect
is trying to read memory allocated by the first thread (thread #1) that
first called mysql_real_connect, 
after that same memory was freed by another thread (thread #2) that
called mysql_server_end as it was 
finishing. Each thread has it's own libdbi instance, so why is
libmysqlclient sharring memory between 
threads? It's looking like the libmysqlclient isn't thread safe.

==2828== Invalid read of size 1
==2828==    at 0x4A3A042: strlen (vg_replace_strmem.c:461)
==2828==    by 0x141A46C1: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A5468: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A54A8: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A373E: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x4FE041E: __pthread_once_slow (pthread_once.c:116)
==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64-
linux-gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux-
gnu/dbd/libdbdmysql.so)
==2828==    by 0x4DC37AE: dbi_conn_connect (in /usr/lib/x86_64-linux-
gnu/libdbi.so.1.1.0)
==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
==2828==    by 0x1357F6: DeleteLogEntry (database.c:440)
==2828==    by 0x131ECE: taskCallProcInThread (tasks.c:73)
==2828==    by 0x4FD7668: start_thread (pthread_create.c:479)
==2828==    by 0x567B322: clone (clone.S:95)

Memory was freed in a thread (lets call it thread #2) that ran and
completed. It created it's own 
dbi instance, did it's thing, and then closed it's MySQL connection and
shutdown the instance.
So why is it freeing memory deep in the mysql driver that it did not
allocate?

==2828==  Address 0x1343e698 is 424 bytes inside a block of size 4,088
free'd
==2828==    at 0x4A37F90: free (vg_replace_malloc.c:540)
==2828==    by 0x141A9B8A: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A8756: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x14148982: mysql_server_end (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x12F4684C: dbd_finalize (in /usr/lib/x86_64-linux-
gnu/dbd/libdbdmysql.so)
==2828==    by 0x4DC2E4D: dbi_shutdown_r (in /usr/lib/x86_64-linux-
gnu/libdbi.so.1.1.0)
==2828==    by 0x134440: dbi_instance_free (database.c:56)
==2828==    by 0x4FD6600: __nptl_deallocate_tsd.part.0
(pthread_create.c:301)
==2828==    by 0x4FD7689: __nptl_deallocate_tsd (pthread_create.c:256)
==2828==    by 0x4FD7689: start_thread (pthread_create.c:490)
==2828==    by 0x567B322: clone (clone.S:95)

This is in the first thread in the program to make use of libdbi/MySQL,
and it is still 
running/connected to MySQL using a unique libdbi intance.

==2828==  Block was alloc'd at
==2828==    at 0x4A36E96: malloc (vg_replace_malloc.c:309)
==2828==    by 0x141A9A30: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A9AEB: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A4E1C: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x1419CFA0: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x1419D95E: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x1416FD29: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A2A66: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A3771: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x4FE041E: __pthread_once_slow (pthread_once.c:116)
==2828==    by 0x141A320D: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A41B2: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141A429E: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x1415472B: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x1415492A: ??? (in /usr/lib/x86_64-linux-
gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x141577A8: mysql_real_connect (in /usr/lib/x86_64-
linux-gnu/libmysqlclient.so.21.1.17)
==2828==    by 0x12F47435: dbd_connect (in /usr/lib/x86_64-linux-
gnu/dbd/libdbdmysql.so)
==2828==    by 0x4DC37AE: dbi_conn_connect (in /usr/lib/x86_64-linux-
gnu/libdbi.so.1.1.0)
==2828==    by 0x134BFA: dbSetupConnection (database.c:211)
==2828==    by 0x13D2F4: GetItemMetaData (database.c:2960)
==2828== 

[libdbi-users] Requiere aditional libraries

[Azael D. <aza...@gm...>]

libpq.so.5: cannot open shared object file: No such file or directory
libdbi: Failed to load driver: /usr/lib/dbd/libdbdpgsql.so

Hi only want to comment this error mesage it is done because the posrgres
library is not install,
I check it installing and uninstalling that library.

I have installed mysql library and  myy program run correctly only this
message is error.

I attach my test code.

[libdbi-users] libdbi-dbd-oracle connection issues

[Evan R. <er...@uv...>]

I have what I think is a working libdbi setup with libdbi-dbd-oracle
built with the Oracle Instant Client 11.2

I can use the perl DBI Oracle package to connect with a connection syntax of

$dbh = DBI->connect("dbi:Oracle:host=$host;port=$port;sid=$sid", $user, 
$pass);

which does NOT require the tnsnames.ora configuration file.

I can't figure out what the libdbi-dbd-oracle connection call syntax 
should be.

The code seems to imply that only a tnsnames dbname can be used, but since
perl seems to be able to connect directly I am a little bit stuck.

What is the connect syntax for libdbi to connect to an oracle database
without requiring a tnsnames.ora file?

Any pointers to documentation or exaples would be appreciated.

Evan

Re: [libdbi-users] Lib DBI windows version

[Simon W. <si...@gi...>]

And you too Markus. Your children should certainly be your priority. Don't feel bad.

A happy heathy new year to you all!

On 2018 12 31 01:37:45 UTC, Pabitra Dash <pab...@ya...> wrote:
>Thank you Simon for your info.
>Regards, 
>Pabitra 
>
>On Monday, 31 December 2018, 5:42:40 AM GMT+5:30, Markus Hoenicka
><mar...@mh...> wrote:  
> 
> 
>Hi,
>
>"dead" is relative here. It is unfortunately pretty much true that I
>was not able to put much of an effort into further development of
>libdbi, although there were a couple of bug fixes in the past couple of
>years. I don't mean to whine, but sometimes life takes unexpected
>turns, and in my case they're 6 and 3 years old now. Needless to say,
>this, and a new position with extended responsibilities, caused a shift
>of priorities that won't do my software projects any good. I'd hate to
>see libdbi die of inactivity, but it is hardly possible to sleep less
>than I did in the past couple of years just to do some extra
>programming. And I apologize if I was not able to tend to questions on
>the lists in a timely fashion.
>
>Regarding the original posters question: you can certainly use libdbi
>on Linux and just about any unixish platform. This includes Cygwin
>(http://www.cygwin.com), a library which provides a Unix-like interface
>to Windows. To the best of my knowledge there is no native Windows port
>available which would allow you to use it in native Windows
>applications. Source code is available from a git repository at
>SourceForge:
>
>https://sourceforge.net/p/libdbi/libdbi/ci/master/tree/
>
>Documentation of the dbi interface is here (the Programmer's Guide is
>what you want):
>
>http://libdbi.sourceforge.net/docs.html
>
> 
>
>regards,
>
>Markus
>
> 
>
>On 2018-12-29 03:25, Simon Walter was heard to say:
>
>Hi Pabitra,
>
>I asked a question on this list in 2015 and there has been no activity
>since then. I didn't get an answer to my question. So I assumed that
>the project is pretty dead. There is support for prepared statements.
>So I ended up using APR DBD. The documentation is alright. Though I did
>find myself digging into the source to understand a few things.
>
>I haven't used it on Windows, but the whole point of the APR is cross
>platform usage.
>
>https://apr.apache.org/docs/apr-util/1.6/group___a_p_r___util___d_b_d.html
>
>Best wishes,
>
>Simon
>
>
>On December 28, 2018 4:28:42 PM UTC, Pabitra Dash via libdbi-users
><lib...@li...> wrote:
>Hello, Good day.I have following queries. -As libdbi provides standard
>C abstraction, Can I use it in Windows and Linux platforms?-From where
>can I get DbiXX detail documentation and source code of the
>same? Regards,Pabitra
>
>_______________________________________________
>libdbi-users mailing list
>lib...@li...
>https://lists.sourceforge.net/lists/listinfo/libdbi-users
>
>
> 
>-- 
>Markus Hoenicka
>http://www.mhoenicka.de
>AQ score 38
>  

-- 
Sent from my mobile device. Please excuse my brevity.

Re: [libdbi-users] Lib DBI windows version

[Pabitra D. <pab...@ya...>]

Thank you Simon for your info.
Regards, 
Pabitra 

    On Monday, 31 December 2018, 5:42:40 AM GMT+5:30, Markus Hoenicka <mar...@mh...> wrote:  
 
 
Hi,

"dead" is relative here. It is unfortunately pretty much true that I was not able to put much of an effort into further development of libdbi, although there were a couple of bug fixes in the past couple of years. I don't mean to whine, but sometimes life takes unexpected turns, and in my case they're 6 and 3 years old now. Needless to say, this, and a new position with extended responsibilities, caused a shift of priorities that won't do my software projects any good. I'd hate to see libdbi die of inactivity, but it is hardly possible to sleep less than I did in the past couple of years just to do some extra programming. And I apologize if I was not able to tend to questions on the lists in a timely fashion.

Regarding the original posters question: you can certainly use libdbi on Linux and just about any unixish platform. This includes Cygwin (http://www.cygwin.com), a library which provides a Unix-like interface to Windows. To the best of my knowledge there is no native Windows port available which would allow you to use it in native Windows applications. Source code is available from a git repository at SourceForge:

https://sourceforge.net/p/libdbi/libdbi/ci/master/tree/

Documentation of the dbi interface is here (the Programmer's Guide is what you want):

http://libdbi.sourceforge.net/docs.html

 

regards,

Markus

 

On 2018-12-29 03:25, Simon Walter was heard to say:

Hi Pabitra,

I asked a question on this list in 2015 and there has been no activity since then. I didn't get an answer to my question. So I assumed that the project is pretty dead. There is support for prepared statements. So I ended up using APR DBD. The documentation is alright. Though I did find myself digging into the source to understand a few things.

I haven't used it on Windows, but the whole point of the APR is cross platform usage.

https://apr.apache.org/docs/apr-util/1.6/group___a_p_r___util___d_b_d.html

Best wishes,

Simon


On December 28, 2018 4:28:42 PM UTC, Pabitra Dash via libdbi-users <lib...@li...> wrote:
Hello, Good day.I have following queries. -As libdbi provides standard C abstraction, Can I use it in Windows and Linux platforms?-From where can I get DbiXX detail documentation and source code of the same? Regards,Pabitra

_______________________________________________
libdbi-users mailing list
lib...@li...
https://lists.sourceforge.net/lists/listinfo/libdbi-users


 
-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38
  

Re: [libdbi-users] Lib DBI windows version

[Markus H. <mar...@mh...>]

 

Hi, 

"dead" is relative here. It is unfortunately pretty much true that I was
not able to put much of an effort into further development of libdbi,
although there were a couple of bug fixes in the past couple of years. I
don't mean to whine, but sometimes life takes unexpected turns, and in
my case they're 6 and 3 years old now. Needless to say, this, and a new
position with extended responsibilities, caused a shift of priorities
that won't do my software projects any good. I'd hate to see libdbi die
of inactivity, but it is hardly possible to sleep less than I did in the
past couple of years just to do some extra programming. And I apologize
if I was not able to tend to questions on the lists in a timely fashion.


Regarding the original posters question: you can certainly use libdbi on
Linux and just about any unixish platform. This includes Cygwin
(http://www.cygwin.com), a library which provides a Unix-like interface
to Windows. To the best of my knowledge there is no native Windows port
available which would allow you to use it in native Windows
applications. Source code is available from a git repository at
SourceForge: 

https://sourceforge.net/p/libdbi/libdbi/ci/master/tree/ 

Documentation of the dbi interface is here (the Programmer's Guide is
what you want): 

http://libdbi.sourceforge.net/docs.html 

regards, 

Markus 

On 2018-12-29 03:25, Simon Walter was heard to say: 

> Hi Pabitra,
> 
> I asked a question on this list in 2015 and there has been no activity since then. I didn't get an answer to my question. So I assumed that the project is pretty dead. There is support for prepared statements. So I ended up using APR DBD. The documentation is alright. Though I did find myself digging into the source to understand a few things.
> 
> I haven't used it on Windows, but the whole point of the APR is cross platform usage.
> 
> https://apr.apache.org/docs/apr-util/1.6/group___a_p_r___util___d_b_d.html [2]
> 
> Best wishes,
> 
> Simon
> 
> On December 28, 2018 4:28:42 PM UTC, Pabitra Dash via libdbi-users <lib...@li...> wrote: 
> 
>> Hello, 
>> 
>> Good day. 
>> I have following queries. 
>> 
>> -As libdbi provides standard C abstraction, Can I use it in Windows and Linux platforms? 
>> -From where can I get DbiXX detail documentation and source code of the same? 
>> 
>> Regards, 
>> Pabitra
> 
> _______________________________________________
> libdbi-users mailing list
> lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libdbi-users [1]

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

 

Links:
------
[1] https://lists.sourceforge.net/lists/listinfo/libdbi-users
[2]
https://apr.apache.org/docs/apr-util/1.6/group___a_p_r___util___d_b_d.html

Re: [libdbi-users] Lib DBI windows version

[Simon W. <si...@gi...>]

Hi Pabitra,

I asked a question on this list in 2015 and there has been no activity since then. I didn't get an answer to my question. So I assumed that the project is pretty dead. There is support for prepared statements. So I ended up using APR DBD. The documentation is alright. Though I did find myself digging into the source to understand a few things.

I haven't used it on Windows, but the whole point of the APR is cross platform usage.

https://apr.apache.org/docs/apr-util/1.6/group___a_p_r___util___d_b_d.html

Best wishes,

Simon


On December 28, 2018 4:28:42 PM UTC, Pabitra Dash via libdbi-users <lib...@li...> wrote:
>Hello,
>Good day.I have following queries.
>-As libdbi provides standard C abstraction, Can I use it in Windows and
>Linux platforms?-From where can I get DbiXX detail documentation and
>source code of the same?
>Regards,Pabitra

[libdbi-users] Lib DBI windows version

[Pabitra D. <pab...@ya...>]

Hello,
Good day.I have following queries.
-As libdbi provides standard C abstraction, Can I use it in Windows and Linux platforms?-From where can I get DbiXX detail documentation and source code of the same?
Regards,Pabitra

Re: [libdbi-users] Problem getting SQLite last sequence value

[Tx0 <tx...@st...>]

On 2016-08-02 00:44, Markus Hoenicka wrote:
> Hi,
> 
> unfortunately I didn't find the time to actually reproduce your
> problem. However, I've found the following conspicuous line in
> dbd_sqlite3.c (line 1236 in my copy):
> 
> || strstr(curr_type, "INTEGER PRIMARY KEY") /* BAD BAD HACK */
> 
> It is quite possible that the "bad bad" part of this hack came to life
> in your example. Could you please test your code again with a slightly
> modified table definition like this:
> 
>    inode integer primary key not null autoincrement,
> 
> Please let me know if this helps.
> 
> regards,
> Markus

Hi Markus,

thanks for your reply. I've wrote a small fragment to do some tests.
I've put it on Gist:

   
https://gist.github.com/StrumentiResistenti/58943f64c7e702dc4f77c15a120619ea

This is the output (I think it's quite self explanatory):

  ---> QUERY: [create table if not exists objects (
     inode integer primary key autoincrement not null,
     objectname text(255) not null,
     last_autotag timestamp not null default current_timestamp,
     checksum text(40) not null default '',
     symlink text(1024) not null default '')]
no tables in statement !
  ---> QUERY: [insert into objects (objectname) values ('testname1'), 
('testname2'), ('testname3')]
no tables in statement !
  ---> QUERY: [select last_insert_rowid()]
no tables in statement !
  <--- callback got a ___STRING___ field: "3"
no tables in statement !
  ---> QUERY: [select cast(last_insert_rowid() as int)]
no tables in statement !
  <--- callback got a ___STRING___ field: "3"
Last inserted sequence number is 3

Clearly there are two problems to solve. The first one is why, inside
Tagsistant, I'm not able to fetch the right number, provided that the
Gist fragment succeeds. The second one is (I suppose) internal to the
DBI driver: why the sequence number is returned as a string instead
of a integer?

I'll address the first one. I hope DBD developers could do something
about the second one.

Btw. at line 1198 of dbd_sqlite3.c there's a printf() call that should
be commented out:

   printf("no tables in statement !\n");

It prints this message every time the dbi_conn_ping() function is called
on SQLite.

Thanks again.
Ciao!
-- 
Tx0 - GNU/Linux hacker
Author of http://www.tagsistant.net/

Re: [libdbi-users] Problem getting SQLite last sequence value

[Markus H. <mar...@mh...>]

At 2016-07-31 17:19, Tx0 was heard to say:
> Hi,
> 
> I'm the author of Tagsistant, a semantic filesystem for Linux,
> and this is my first post on this list, so I hope this is the
> right place to ask and to not infringe any rule.
> 
> Tagsistant uses a SQL database as its backend, and libDBI to
> abstract the interaction with different RDBMS. Currently, MySQL
> and SQLite are supported. Before using DBI, the SQLite  API
> was directly used.
> 
> I had no problems with both RDBMS up to DBI version 0.8.x.
> With 0.9.x I'm facing a strange behavior in getting the last
> inserted primary key from a table, but with the SQLite backend
> only.
> 
> This is the schema of the objects table in SQLite:
> 
> CREATE TABLE objects (
>    inode integer not null primary key autoincrement,
>    objectname text(255) not null,
>    last_autotag timestamp not null default current_timestamp,
>    checksum text(40) not null default '',
>    symlink text(1024) not null default ''
> );
> 
> If I insert something from the sqlite3 shell, everything works
> fine:
> 
> sqlite> select max(inode) from objects;
> 3
> sqlite> insert into objects (objectname) values ('testobj');
> sqlite> select last_insert_rowid();
> 4
> sqlite> select max(inode) from objects;
> 4
> sqlite>
> 
> When the insert is done inside Tagsistant, the new record properly
> enters the objects table. After that, Tagsistant uses the code
> below to read the last inserted primary key.
> 
> typedef uint32_t tagsistant_inode
> 
> tagsistant_inode
> tagsistant_last_insert_id(dbi_conn conn)
> {
> #if TAGSISTANT_USE_INTERNAL_SEQUENCES
>      tagsistant_inode inode = 0;
> 
>      switch (tagsistant.sql_database_driver) {
>          case TAGSISTANT_DBI_SQLITE_BACKEND:
>              tagsistant_query(
>                  "SELECT cast(last_insert_rowid() as int)",
>                  conn,
>                  tagsistant_return_integer,
>                  &inode);
>              break;
> 
>          case TAGSISTANT_DBI_MYSQL_BACKEND:
>              tagsistant_query(
>                  "SELECT last_insert_id()",
>                  conn,
>                  tagsistant_return_integer,
>                  &inode);
>              break;
>      }
> 
>      return (inode);
> #else
>      return(dbi_conn_sequence_last(conn, NULL));
> #endif
> }
> 
> If I set TAGSISTANT_USE_INTERNAL_SEQUENCES, no useful value is returned
> by dbi_conn_sequence_last(). It's always 1.
> 
> And here is the callback used to load the value, if my custom code is
> used in place of dbi_conn_sequence_last(). Note that the value is not
> returned as an integer, as expected. It's instead a DBI_TYPE_STRING and
> it's always "1".
> 
> int tagsistant_return_integer(void *return_integer, dbi_result result)
> {
>      uint32_t *buffer = (uint32_t *) return_integer;
>      *buffer = 0;
> 
>      unsigned int type = dbi_result_get_field_type_idx(result, 1);
>      if (type == DBI_TYPE_INTEGER) {
>          unsigned int size = dbi_result_get_field_attribs_idx(result, 
> 1);
>          unsigned int is_unsigned = size & DBI_INTEGER_UNSIGNED;
>          size = size & DBI_INTEGER_SIZEMASK;
>          switch (size) {
>              case DBI_INTEGER_SIZE8:
>                  if (is_unsigned)
>                      *buffer = dbi_result_get_ulonglong_idx(result, 1);
>                  else
>                      *buffer = dbi_result_get_longlong_idx(result, 1);
>                  break;
>              case DBI_INTEGER_SIZE4:
>              case DBI_INTEGER_SIZE3:
>                  if (is_unsigned)
>                      *buffer = dbi_result_get_uint_idx(result, 1);
>                  else
>                      *buffer = dbi_result_get_int_idx(result, 1);
>                  break;
>              case DBI_INTEGER_SIZE2:
>                  if (is_unsigned)
>                      *buffer = dbi_result_get_ushort_idx(result, 1);
>                  else
>                      *buffer = dbi_result_get_short_idx(result, 1);
>                  break;
>              case DBI_INTEGER_SIZE1:
>                  if (is_unsigned)
>                      *buffer = dbi_result_get_uchar_idx(result, 1);
>                  else
>                      *buffer = dbi_result_get_char_idx(result, 1);
>                  break;
>          }
>      } else if (type == DBI_TYPE_STRING) {
>          const gchar *int_string = dbi_result_get_string_idx(result, 
> 1);
>          *buffer = atoi(int_string);
>          dbg('s', LOG_INFO, "tagsistant_return_integer called on string
> field %s", int_string);
>      }
> 
>      dbg('s', LOG_INFO, "Returning integer: %d", *buffer);
> 
>      return (0);
> }
> 
> Could this behavior relate to the dbi_conn reference passed? But then,
> why
> it works with MySQL and not with SQLite?
> 
> Or maybe it's the table schema? Then why everything works fine from the
> sqlite3 shell?
> 
> Thank you very much.
> Ciao!

Hi,

unfortunately I didn't find the time to actually reproduce your problem. 
However, I've found the following conspicuous line in dbd_sqlite3.c 
(line 1236 in my copy):

|| strstr(curr_type, "INTEGER PRIMARY KEY") /* BAD BAD HACK */

It is quite possible that the "bad bad" part of this hack came to life 
in your example. Could you please test your code again with a slightly 
modified table definition like this:

    inode integer primary key not null autoincrement,

Please let me know if this helps.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[libdbi-users] Problem getting SQLite last sequence value

[Tx0 <tx...@st...>]

Hi,

I'm the author of Tagsistant, a semantic filesystem for Linux,
and this is my first post on this list, so I hope this is the
right place to ask and to not infringe any rule.

Tagsistant uses a SQL database as its backend, and libDBI to
abstract the interaction with different RDBMS. Currently, MySQL
and SQLite are supported. Before using DBI, the SQLite  API
was directly used.

I had no problems with both RDBMS up to DBI version 0.8.x.
With 0.9.x I'm facing a strange behavior in getting the last
inserted primary key from a table, but with the SQLite backend
only.

This is the schema of the objects table in SQLite:

CREATE TABLE objects (
   inode integer not null primary key autoincrement,
   objectname text(255) not null,
   last_autotag timestamp not null default current_timestamp,
   checksum text(40) not null default '',
   symlink text(1024) not null default ''
);

If I insert something from the sqlite3 shell, everything works
fine:

sqlite> select max(inode) from objects;
3
sqlite> insert into objects (objectname) values ('testobj');
sqlite> select last_insert_rowid();
4
sqlite> select max(inode) from objects;
4
sqlite>

When the insert is done inside Tagsistant, the new record properly
enters the objects table. After that, Tagsistant uses the code
below to read the last inserted primary key.

typedef uint32_t tagsistant_inode

tagsistant_inode
tagsistant_last_insert_id(dbi_conn conn)
{
#if TAGSISTANT_USE_INTERNAL_SEQUENCES
     tagsistant_inode inode = 0;

     switch (tagsistant.sql_database_driver) {
         case TAGSISTANT_DBI_SQLITE_BACKEND:
             tagsistant_query(
                 "SELECT cast(last_insert_rowid() as int)",
                 conn,
                 tagsistant_return_integer,
                 &inode);
             break;

         case TAGSISTANT_DBI_MYSQL_BACKEND:
             tagsistant_query(
                 "SELECT last_insert_id()",
                 conn,
                 tagsistant_return_integer,
                 &inode);
             break;
     }

     return (inode);
#else
     return(dbi_conn_sequence_last(conn, NULL));
#endif
}

If I set TAGSISTANT_USE_INTERNAL_SEQUENCES, no useful value is returned
by dbi_conn_sequence_last(). It's always 1.

And here is the callback used to load the value, if my custom code is
used in place of dbi_conn_sequence_last(). Note that the value is not
returned as an integer, as expected. It's instead a DBI_TYPE_STRING and
it's always "1".

int tagsistant_return_integer(void *return_integer, dbi_result result)
{
     uint32_t *buffer = (uint32_t *) return_integer;
     *buffer = 0;

     unsigned int type = dbi_result_get_field_type_idx(result, 1);
     if (type == DBI_TYPE_INTEGER) {
         unsigned int size = dbi_result_get_field_attribs_idx(result, 1);
         unsigned int is_unsigned = size & DBI_INTEGER_UNSIGNED;
         size = size & DBI_INTEGER_SIZEMASK;
         switch (size) {
             case DBI_INTEGER_SIZE8:
                 if (is_unsigned)
                     *buffer = dbi_result_get_ulonglong_idx(result, 1);
                 else
                     *buffer = dbi_result_get_longlong_idx(result, 1);
                 break;
             case DBI_INTEGER_SIZE4:
             case DBI_INTEGER_SIZE3:
                 if (is_unsigned)
                     *buffer = dbi_result_get_uint_idx(result, 1);
                 else
                     *buffer = dbi_result_get_int_idx(result, 1);
                 break;
             case DBI_INTEGER_SIZE2:
                 if (is_unsigned)
                     *buffer = dbi_result_get_ushort_idx(result, 1);
                 else
                     *buffer = dbi_result_get_short_idx(result, 1);
                 break;
             case DBI_INTEGER_SIZE1:
                 if (is_unsigned)
                     *buffer = dbi_result_get_uchar_idx(result, 1);
                 else
                     *buffer = dbi_result_get_char_idx(result, 1);
                 break;
         }
     } else if (type == DBI_TYPE_STRING) {
         const gchar *int_string = dbi_result_get_string_idx(result, 1);
         *buffer = atoi(int_string);
         dbg('s', LOG_INFO, "tagsistant_return_integer called on string 
field %s", int_string);
     }

     dbg('s', LOG_INFO, "Returning integer: %d", *buffer);

     return (0);
}

Could this behavior relate to the dbi_conn reference passed? But then, 
why
it works with MySQL and not with SQLite?

Or maybe it's the table schema? Then why everything works fine from the
sqlite3 shell?

Thank you very much.
Ciao!
-- 
Tx0
Author of http://www.tagsistant.net/

Re: [libdbi-users] prepared statements

[Simon W. <si...@gi...>]

On 06/12/2015 01:07 AM, Alex Bligh wrote:
> 
> On 11 Jun 2015, at 12:00, Simon Walter <si...@gi...> wrote:
> 
>> Looking at projects that use APR, many seem to include the entire thing
>> in their code base. I'm guessing that one of the drawbacks is that APR
>> comes with a lot of baggage so to speak.
> 
> You can dynamically link to an APR library.
> 
>> Have you used both libdbi and the database bits of APR? Is there any
>> advantage to using libdbi compared to APR?
> 
> I've either used one or the other. I expect you can use both but
> can't see any reason to.
> 
> apr is more frequently updated, has many other functions (e.g.
> a complete memory pool manager), and works on a wider range of
> platforms I believe (including Windows).
> 
> libdbi is simpler than APR to get used to, and has a smaller memory
> footprint.
> 
> You should also compare the choice of backends if you care about
> that.
> 

OK, that's very helpful. Thank you. I'm not sure I want to change my
code around too much until I have more tests written. It's good to know
there is another usable option.

Re: [libdbi-users] prepared statements

[Markus H. <mar...@mh...>]

At 2015-06-11 09:12, Simon Walter was heard to say:
> Hi all,
> 
> I'm new to using libdbi and C in general. I am wondering why libdbi has
> no support for prepared statements.
> 
> Is it that because the way libdbi is written, a prepared statement 
> would
> not provide any benefit? Or is it just lacking that feature? If it's
> just lacking the feature, are there plans to add that?
> 
> If the application I'm writing can benefit from prepared statements, I
> would like to eventually either add that feature to libdbi or I may 
> have
> to use something else in the future.
> 
> I haven't taken a deep look at the libdbi code. So any insight is
> appreciated.
> 
> Kind regards,
> 
> Simon
> 
> PS
> 
> If I should have sent this to the development list, please let me know.
> Thanks.

Hi,

I can't tell if prepared statements would speed up things when accessing 
databases through libdbi. As in many free software projects, the main 
reason for the lack of prepared statements is that nobody invested his 
time in implementing them. This includes myself, although I can't recall 
many requests for this feature. However, I think libdbi could support 
prepared statements if someone takes care of it. My commitments to my 
dayjob and to my family do not leave much dev time currently.

If you think about helping to implement this feature, you need to 
consider both libdbi and libdbi-drivers. libdbi needs a generic 
interface to prepared statements that covers all database backends. The 
drivers have to implement the feature using the functions of their 
respective client libraries. Therefore, it is best to design and discuss 
the interface first, and then to move on from there.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [libdbi-users] prepared statements

[Alex B. <al...@al...>]

On 11 Jun 2015, at 12:00, Simon Walter <si...@gi...> wrote:

> Looking at projects that use APR, many seem to include the entire thing
> in their code base. I'm guessing that one of the drawbacks is that APR
> comes with a lot of baggage so to speak.

You can dynamically link to an APR library.

> Have you used both libdbi and the database bits of APR? Is there any
> advantage to using libdbi compared to APR?

I've either used one or the other. I expect you can use both but
can't see any reason to.

apr is more frequently updated, has many other functions (e.g.
a complete memory pool manager), and works on a wider range of
platforms I believe (including Windows).

libdbi is simpler than APR to get used to, and has a smaller memory
footprint.

You should also compare the choice of backends if you care about
that.

-- 
Alex Bligh

Re: [libdbi-users] prepared statements

[Simon W. <si...@gi...>]

On 06/11/2015 06:07 PM, Alex Bligh wrote:
> 
> On 11 Jun 2015, at 08:12, Simon Walter <si...@gi...> wrote:
> 
>> If the application I'm writing can benefit from prepared statements, I
>> would like to eventually either add that feature to libdbi or I may have
>> to use something else in the future.
> 
> Not much use I know, but I can confirm it doesn't support prepared
> statement currently. You might consider APR, i.e.
> 
> http://apr.apache.org/docs/apr/2.0/group___a_p_r___util___d_b_d.html
> 


That looks like an awful lot of nice C code. Thank you for the tip!

Looking at projects that use APR, many seem to include the entire thing
in their code base. I'm guessing that one of the drawbacks is that APR
comes with a lot of baggage so to speak.

Have you used both libdbi and the database bits of APR? Is there any
advantage to using libdbi compared to APR?

Simon

Re: [libdbi-users] prepared statements

[Alex B. <al...@al...>]

On 11 Jun 2015, at 08:12, Simon Walter <si...@gi...> wrote:

> If the application I'm writing can benefit from prepared statements, I
> would like to eventually either add that feature to libdbi or I may have
> to use something else in the future.

Not much use I know, but I can confirm it doesn't support prepared
statement currently. You might consider APR, i.e.

http://apr.apache.org/docs/apr/2.0/group___a_p_r___util___d_b_d.html

-- 
Alex Bligh

[libdbi-users] prepared statements

[Simon W. <si...@gi...>]

Hi all,

I'm new to using libdbi and C in general. I am wondering why libdbi has
no support for prepared statements.

Is it that because the way libdbi is written, a prepared statement would
not provide any benefit? Or is it just lacking that feature? If it's
just lacking the feature, are there plans to add that?

If the application I'm writing can benefit from prepared statements, I
would like to eventually either add that feature to libdbi or I may have
to use something else in the future.

I haven't taken a deep look at the libdbi code. So any insight is
appreciated.

Kind regards,

Simon

PS

If I should have sent this to the development list, please let me know.
Thanks.

Re: [libdbi-users] Valgrind error with pgsql: Conditional jump or move depends on uninitialised value(s)

[Alex B. <al...@al...>]

On 4 Nov 2014, at 12:09, Markus Hoenicka <mar...@mh...> wrote:

> At 2014-11-04 11:41 Alex Bligh was heard to say:
>> On 3 Nov 2014, at 22:54, Markus Hoenicka <mar...@mh...> wrote:
>>> I wouldn't bet from valgrind's output that it is libdbi variables which are uninitialized. Can you re-run your test with a different database engine? I'd suggest using the sqlite3 driver as this engine has few if any external dependencies.
>> I reran with mysql and it doesn't appear. I presume it's the pgsql dbi driver.
> 
> This is one explanation. Another explanation is that the PostgreSQL client library or one of the libraries it depends on causes these messages. Could you please fire up the psql command line utility under valgrind and do what your test program does, i.e. establish a connection? If this test does not report unitialized variables, we'll have to revisit the pgsql driver.

The valgrind nastiness appears even without the correct auth credentials.

Running using psql is very odd:

$ valgrind psql --username x -W 127.0.0.1
==14382== Memcheck, a memory error detector
==14382== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==14382== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==14382== Command: /usr/bin/psql --username x -W 127.0.0.1
==14382==
Password for user x:
psql: FATAL:  Peer authentication failed for user "x"

No valgrind errors, but no valgrind summary either. Not quite sure what causes that.

-- 
Alex Bligh

Re: [libdbi-users] Valgrind error with pgsql: Conditional jump or move depends on uninitialised value(s)

[Markus H. <mar...@mh...>]

At 2014-11-04 11:41 Alex Bligh was heard to say:
> On 3 Nov 2014, at 22:54, Markus Hoenicka <mar...@mh...> 
> wrote:
> 
>> I wouldn't bet from valgrind's output that it is libdbi variables 
>> which are uninitialized. Can you re-run your test with a different 
>> database engine? I'd suggest using the sqlite3 driver as this engine 
>> has few if any external dependencies.
> 
> I reran with mysql and it doesn't appear. I presume it's the pgsql dbi 
> driver.

This is one explanation. Another explanation is that the PostgreSQL 
client library or one of the libraries it depends on causes these 
messages. Could you please fire up the psql command line utility under 
valgrind and do what your test program does, i.e. establish a 
connection? If this test does not report unitialized variables, we'll 
have to revisit the pgsql driver.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [libdbi-users] Valgrind error with pgsql: Conditional jump or move depends on uninitialised value(s)

[Alex B. <al...@al...>]

On 3 Nov 2014, at 22:54, Markus Hoenicka <mar...@mh...> wrote:

> I wouldn't bet from valgrind's output that it is libdbi variables which are uninitialized. Can you re-run your test with a different database engine? I'd suggest using the sqlite3 driver as this engine has few if any external dependencies.

I reran with mysql and it doesn't appear. I presume it's the pgsql dbi driver.

-- 
Alex Bligh

Re: [libdbi-users] Valgrind error with pgsql: Conditional jump or move depends on uninitialised value(s)

[Markus H. <mar...@mh...>]

At 2014-11-02 19:05 Alex Bligh was heard to say:
> I have a very simple program (below) only marginally adapted from the
> example here:
>   
> http://libdbi.sourceforge.net/docs/programmers-guide/quickstart-code.html
> 
> which works, but when run through valgrind produces an enormous number
> of errors.
> The program in its entirety is below. It makes a connection to pgsql, 
> does no
> queries, and immediately shuts the connection.
> 
> I'm not so much worried about the alleged memory leaks but the 85 
> instances of
> "Conditional jump or move depends on uninitialised value(s)". I don't
> know if this is a real problem, but even if not the challenge here
> is that /any/ query via dbi is now tainted as uninitialised, so it's 
> impossible
> to see 'real' errors.
> 
> I've not attached a full valgrind dump as it's enormous. However, you 
> can see
> the first two below.
> 
> Any ideas?
> 

Hi,

I wouldn't bet from valgrind's output that it is libdbi variables which 
are uninitialized. Can you re-run your test with a different database 
engine? I'd suggest using the sqlite3 driver as this engine has few if 
any external dependencies.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38