libdbi-users — General libdbi discussion and support

Re: [libdbi-users] libdbi+mysql speed issues

[Adam J. M. <lib...@ad...>]

Hi Markus,

I definitely can, anecdotal evidence suggests to me that this issue does not occur on postgres (I use rrdtool+postgres similarly and havne't seen the issue), but I'll make some actual tests and report back.

-Adam


On Jul 9, 2011, at 8:39 AM, mar...@mh... wrote:

> Adam Jacob Muller writes:
>> Hi,
>> I'm developing an application that utilizes rrdtool (and the lidbdi interface into rrdtool) heavily and having some performance issues that seem to center around libdbi,
>> 
> 
> Hi Adam,
> 
> I can't see any obvious issues with your C code, so I assume the speed
> differences really stem from libdbi, one way or another. Unfortunately
> I currently lack the time for any deeper analysis, but I'm somewhat
> surprised about your results. libdbi has originally been developed as
> a wrapper for MySQL and PostgreSQL and should work fairly well with
> these database engines (other engines are a *lot* harder to fit). The
> only overhead that the mysql driver contains is the column type
> analysis and data conversion stuff. Some other engines have to buffer
> the data and thus create way more overhead.
> 
> Did you have a chance to verify your results with any of the other
> database drivers?
> 
> regards,
> Markus
> 
> -- 
> Markus Hoenicka
> http://www.mhoenicka.de
> AQ score 38
> 
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security 
> threats, fraudulent activity, and more. Splunk takes this data and makes 
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> libdbi-users mailing list
> lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libdbi-users

[libdbi-users] libdbi+mysql speed issues

[<mar...@mh...>]

Adam Jacob Muller writes:
 > Hi,
 > I'm developing an application that utilizes rrdtool (and the lidbdi interface into rrdtool) heavily and having some performance issues that seem to center around libdbi,
 > 

Hi Adam,

I can't see any obvious issues with your C code, so I assume the speed
differences really stem from libdbi, one way or another. Unfortunately
I currently lack the time for any deeper analysis, but I'm somewhat
surprised about your results. libdbi has originally been developed as
a wrapper for MySQL and PostgreSQL and should work fairly well with
these database engines (other engines are a *lot* harder to fit). The
only overhead that the mysql driver contains is the column type
analysis and data conversion stuff. Some other engines have to buffer
the data and thus create way more overhead.

Did you have a chance to verify your results with any of the other
database drivers?

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[libdbi-users] libdbi+mysql speed issues

[Adam J. M. <lib...@ad...>]

Hi,
I'm developing an application that utilizes rrdtool (and the lidbdi interface into rrdtool) heavily and having some performance issues that seem to center around libdbi,

# ./dbi
dbi_conn_queryf took 0.5866
dbi_result_next_row took 15.4186
looped through 71891 rows

# ./mysql 
MySQL client version: 5.1.56
mysql_query took 0.0021
mysql_fetch_row took 0.5584
looped through 71891 rows

libdbi here appears to simply be spectacularly slow, I assume its something that I/the rrdtool libdbi are doing wrong. source for dbi.c and mysql.c follows (yes, i'm a horrible c programmer, I know!).


Thanks,

-Adam


# cat dbi.c 
#include <stdio.h>
#include <dbi/dbi.h>
#include <sys/time.h>

int main() {
  dbi_conn conn;
  dbi_result result;

  struct timeval query_start;
  struct timeval query_end;
  struct timeval query_elapsed;
  float real_query_elapsed;

  struct timeval fetch_start;
  struct timeval fetch_end;
  struct timeval fetch_elapsed;
  float real_fetch_elapsed=0;
  int rowcount;

  dbi_initialize(NULL);
  conn = dbi_conn_new("mysql");
  dbi_conn_set_option(conn, "host", "something");
  dbi_conn_set_option(conn, "username", "something");
  dbi_conn_set_option(conn, "password", "something");
  dbi_conn_set_option(conn, "dbname", "something");
  dbi_conn_set_option(conn, "encoding", "UTF-8");
  if (dbi_conn_connect(conn) < 0) {
      printf("Could not connect. Please check the option settings\n");
      return;
  }
  gettimeofday(&query_start,NULL);
  result = dbi_conn_queryf(conn,"SELECT dtime as rrd_time, counter as rrd_value FROM ifInOctets WHERE id=1920 AND '2011-06-01 00:00:00' < dtime AND dtime < '2011-06-26 00:00:00' ORDER BY dtime");
  gettimeofday(&query_end,NULL);
  timersub(&query_end,&query_start,&query_elapsed);
  real_query_elapsed=(float)query_elapsed.tv_sec;
  real_query_elapsed+=(float)((float)query_elapsed.tv_usec/1000000);
  printf("dbi_conn_queryf took %.4f\n",real_query_elapsed);


  gettimeofday(&fetch_start,NULL);
  rowcount=0;
  if (result) {
	while (dbi_result_next_row(result)) {
		rowcount++;
	}
  }
  gettimeofday(&fetch_end,NULL);
  timersub(&fetch_end,&fetch_start,&fetch_elapsed);
  real_fetch_elapsed=(float)fetch_elapsed.tv_sec;
  real_fetch_elapsed+=(float)((float)fetch_elapsed.tv_usec/1000000);
  printf("dbi_result_next_row took %.4f\n",real_fetch_elapsed);
  printf("looped through %d rows\n",rowcount);
  sleep(30);
  return;
}

# cat mysql.c 
#include <my_global.h>
#include <mysql.h>

int main(int argc, char **argv)
{
MYSQL *conn;
MYSQL_RES *result;
MYSQL_ROW row;
int num_fields;
int i;
  struct timeval query_start;
  struct timeval query_end;
  struct timeval query_elapsed;
  float real_query_elapsed;

  struct timeval fetch_start;
  struct timeval fetch_end;
  struct timeval fetch_elapsed;
  float real_fetch_elapsed=0;

printf("MySQL client version: %s\n", mysql_get_client_info());
conn = mysql_init(NULL);
if (conn == NULL) {
	printf("Error %u: %s\n", mysql_errno(conn), mysql_error(conn));
	exit(1);
}

if (mysql_real_connect(conn, "this", "that", "theotherthing", "redacted", 0, NULL, 0) == NULL) {
	printf("Error %u: %s\n", mysql_errno(conn), mysql_error(conn));
	exit(1);
}

gettimeofday(&query_start,NULL);
mysql_query(conn,"SELECT dtime as rrd_time, counter as rrd_value FROM ifInOctets WHERE id=1920 AND '2011-06-01 00:00:00' < dtime AND dtime < '2011-06-26 00:00:00' ORDER BY dtime");
result=mysql_use_result(conn);
gettimeofday(&query_end,NULL);
timersub(&query_end,&query_start,&query_elapsed);
real_query_elapsed=(float)query_elapsed.tv_sec;
real_query_elapsed+=(float)((float)query_elapsed.tv_usec/1000000);
printf("mysql_query took %.4f\n",real_query_elapsed);

gettimeofday(&fetch_start,NULL);
i=0;
while ((row = mysql_fetch_row(result))) {
	i++;
}
gettimeofday(&fetch_end,NULL);
timersub(&fetch_end,&fetch_start,&fetch_elapsed);
real_fetch_elapsed=(float)fetch_elapsed.tv_sec;
real_fetch_elapsed+=(float)((float)fetch_elapsed.tv_usec/1000000);
printf("mysql_fetch_row took %.4f\n",real_fetch_elapsed);

printf("looped through %d rows\n",i);
}

Re: [libdbi-users] PATCH: sqlite3 parser bugs

[<mar...@mh...>]

Vikram Ambrose writes:
 > And here is an update to the patch for a third bug.

Better late than never ... I've applied your patch, and things seem to
work ok. Thanks for being this patient.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [libdbi-users] sqlite3 multiple tables

[<mar...@mh...>]

[Vikram's original mail from 2010-12-25 got lost from my inbox, so I
have to apologize for (1) being darn late and for (2) not keeping the
thread intact]

 > 1. Artist (id,name)
 > 2. Album (id,VA,name)
 > 3. Album_Artist (artist_id,album_id)
[...]
 > no tables in statement !
 > no tables in statement !

How did you create the tables? Did you specify types for each column?

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[libdbi-users] dbi_conn_close == dbi_conn_destroy?

[Daniel O'N. <do...@ri...>]

I'm curious if there's a way to disconnect a dbi_conn without destroying the 
handle in the process?

The Ferite Libdbi interface works on a 'connect or die' system, something like 
this:

object dbh = Dbi.connect("mysql", "localhost", "root");
if( !dbh ) { // Failed to connect } else { // Connected }

At the moment, I am attempting to modify this to allow, for example:

object dbh = new Dbi();
dbh.connect("mysql", "localhost", "root");
if( !dbh.connected() ) { println( dbh.errmsg() ); }

Then, freely dbh.close() and dbh.open() later on.  The trouble is I can't 
figure out a way to close the connection without destroying the underlying 
dbi_conn handle.

If there isn't currently a way, would a patch implementing, for example, a 
"dbi_conn_disconnect(dbi_conn Conn)" be considered?

[libdbi-users] pydbi - Python bindings for libdbi

[Vikram A. <noe...@gm...>]

Hello List,

Is there any interest in writing python bindings for libDBI?

I've been pondering over the idea of a pyDBI for a while now. The 
advantages of a scriptable interface to libDBI should be obvious. But it 
goes without saying that having such a binding could make for some very 
versatile Web apps.

In combination with projects like Pyro ( 
http://www.xs4all.nl/~irmen/pyro3/ ), (Java RMI equivalent for Python) 
one could have a very powerful network capable database independent SQL 
interface. A layman's ODBC. This would in essence allow anyone to create 
a "libdbi server" of sorts. Accessible from any desktop or web server 
that has python support.

I'm planning on hacking away at it in my free time. libDBI is a very 
straightforward library and as such, I plan to keep my pydbi simple as 
well. Not that I have much python expertise to complicate matters in the 
first place. An almost straight pass through from C to Python, with 
cython taking care of most data type conversions.

Here is the git repo I'm using:
git clone git://pydbi.git.sourceforge.net/gitroot/pydbi/pydbi

Here is a link the git web interface if you do not have git installed:
http://pydbi.git.sourceforge.net/git/gitweb.cgi?p=pydbi/pydbi;a=tree

There is a little test program called test.py in the repo you can try out.

You will need the Python dev files and cython to build the pydbi module. 
Both available from distro repositories. cython is only a compile time 
dependency

$ apt-get install python2.6-dev cython


Cheers,


Vikram.

[libdbi-users] Hello

[<mar...@mh...>]

Vikram Ambrose writes:
 > I've been emailing the list for about a month now with my patches and I 
 > have got no responses. Can someone please confirm that these messages 
 > are going through?

Hi,

they do, and I sincerely apologize that I haven't been able to take
care of your messages and patches yet. My dayjob currently eats all
available time (lest I stop sleeping entirely). I can't make any
promises, but I'll do my very best to return to my libdbi duties asap.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[libdbi-users] Hello

[Vikram A. <noe...@gm...>]

I've been emailing the list for about a month now with my patches and I 
have got no responses. Can someone please confirm that these messages 
are going through?

Re: [libdbi-users] dynamic loading

[Markus H. <mar...@mh...>]

Rainer Gerhards <rge...@hq...> was heard to say:

> I don't remember if you use libtool. If not, Samuel may consider modifying
> libdbi to utilize libtool, which what should not be too hard. As of my
> understanding (never actually tried it), libtool provides full flexibility in
> deciding if dynamic or static linking is used.
>

Dynamic vs. static linking is a different issue. At this time, libdbi  
is not linked against the drivers at all, for the reasons mentioned in  
my previous mail. Instead it uses dlopen() to load the drivers at  
runtime. libtool is involved here only on platforms which do not  
provide their own dlopen() implementation. However, building libdbi as  
a dynamic library uses libtool anyway, and it should be possible to  
statically link a project against libdbi if desired without further  
modifications.

regards,
Markus



-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [libdbi-users] dynamic loading

[Rainer G. <rge...@hq...>]

I don't remember if you use libtool. If not, Samuel may consider modifying
libdbi to utilize libtool, which what should not be too hard. As of my
understanding (never actually tried it), libtool provides full flexibility in
deciding if dynamic or static linking is used.

Rainer

> -----Original Message-----
> From: Markus Hoenicka [mailto:mar...@mh...]
> Sent: Friday, January 14, 2011 12:57 AM
> To: lib...@li...
> Subject: Re: [libdbi-users] dynamic loading
> 
> Quoting "DeLuca, Samuel Louis" <sam...@Va...>:
> 
> > Hello,
> >
> > Is there any way of compiling libdbi without a dynamic loader?  If
> > not, is there any reason why it would be a bad idea for me to write
> > a patch to make it possible to dynamically link in drivers instead
> > of using dynamic loading?
> 
> Hi,
> 
> at this time there is no such option. The benefit of dlopen()ing the
> drivers instead of linking them in is that you don't have to recompile
> libdbi in order to install drivers. This makes it convenient for
> package or port systems to ship individual drivers separately from the
> library. I can't judge though whether your idea is technically
> feasible.
> 
> regards,
> Markus
> 
> 
> --
> Markus Hoenicka
> http://www.mhoenicka.de
> AQ score 38
> 
> 
> 
> -----------------------------------------------------------------------
> -------
> Protect Your Site and Customers from Malware Attacks
> Learn about various malware tactics and how to avoid them. Understand
> malware threats, the impact they can have on your business, and how you
> can protect your company and customers by using code signing.
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> libdbi-users mailing list
> lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libdbi-users

Re: [libdbi-users] dynamic loading

[Markus H. <mar...@mh...>]

Quoting "DeLuca, Samuel Louis" <sam...@Va...>:

> Hello,
>
> Is there any way of compiling libdbi without a dynamic loader?  If  
> not, is there any reason why it would be a bad idea for me to write  
> a patch to make it possible to dynamically link in drivers instead  
> of using dynamic loading?

Hi,

at this time there is no such option. The benefit of dlopen()ing the  
drivers instead of linking them in is that you don't have to recompile  
libdbi in order to install drivers. This makes it convenient for  
package or port systems to ship individual drivers separately from the  
library. I can't judge though whether your idea is technically feasible.

regards,
Markus


-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[libdbi-users] dynamic loading

[DeLuca, S. L. <sam...@Va...>]

Hello, 

Is there any way of compiling libdbi without a dynamic loader?  If not, is there any reason why it would be a bad idea for me to write a patch to make it possible to dynamically link in drivers instead of using dynamic loading? 

Thank you,
Sam DeLuca

Re: [libdbi-users] [syslog-ng] Broken Debug Behavior in libdbi

[Balazs S. <ba...@ba...>]

On Wed, 2011-01-05 at 18:03 -0800, Matthew Hall wrote:
> On Tue, Jan 04, 2011 at 06:20:11PM -0800, Matthew Hall wrote:
> > 3) To work around these problems for now I'm going to patch configure.in 
> > and rerun autoconf unless there is an alternative proposal.
> > 
> > Thanks,
> > Matthew Hall.
> 
> As promised, a hacky workaround patch is pasted below.
> 
> I'd like to improve upon this bad situation, probably by appending the 
> user CFLAGS after the custom package CFLAGS to take priority (the last 
> entries generally take precedence, AIUI), but I'm not incredibly 
> experienced with how this code is supposed to work so I'd like some 
> advice before trying something like this.

autoconf automatically adds values to variables like CFLAGS (like it
automatically adds -O2 if the user doesn't specify it in the
environment). Since this is the default behaviour, packages often
completely wipe out the CFLAGS value in their debug-builds in order to
avoid -O2 being present.

The solution in my configure scripts is to use the shell variable
"ac_cv_env_CFLAGS_value", which stores the original user-sipplied value.

This way, those don't get ignore in debug builds, but I can strip out
-O2 unless explicitly specified by the user.

-- 
Bazsi

Re: [libdbi-users] [syslog-ng] Broken Debug Behavior in libdbi

[Matthew H. <mh...@mh...>]

On Tue, Jan 04, 2011 at 06:20:11PM -0800, Matthew Hall wrote:
> 3) To work around these problems for now I'm going to patch configure.in 
> and rerun autoconf unless there is an alternative proposal.
> 
> Thanks,
> Matthew Hall.

As promised, a hacky workaround patch is pasted below.

I'd like to improve upon this bad situation, probably by appending the 
user CFLAGS after the custom package CFLAGS to take priority (the last 
entries generally take precedence, AIUI), but I'm not incredibly 
experienced with how this code is supposed to work so I'd like some 
advice before trying something like this.

Thanks,
Matthew.

diff -ruN -x '*m4*' -x 'config.*' -x configure -x 'Makefile.*' -x ltmain.sh old/libdbi-0.8.4/configure.in new/libdbi-0.8.4/configure.in
--- old/libdbi-0.8.4/configure.in	2010-08-31 15:35:12.000000000 -0700
+++ new/libdbi-0.8.4/configure.in	2011-01-05 13:36:54.561314455 -0800
@@ -47,25 +47,25 @@
                 CFLAGS="-xO4 -fast -w -fsimple -native -xcg92"
                 PROFILE="-v -xpg -g -xO4 -fast -native -fsimple -xcg92 -Dsuncc" ;;
         *)
-                DEBUG="-g"
-                CFLAGS="-O"
-                PROFILE="-g -p" ;;
+                DEBUG="-g -m32 -I /home/y/include"
+                CFLAGS="-O -m32 -I /home/y/include"
+                PROFILE="-g -p -m32 -I /home/y/include" ;;
         esac
 else
 
         case $host in
         *-*-linux*)
-                DEBUG="-g -Wall -D_REENTRANT -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
-                CFLAGS="-O20 -ffast-math -D_REENTRANT -fsigned-char -std=gnu99"
-                PROFILE="-pg -g -O20 -ffast-math -D_REENTRANT -fsigned-char -std=gnu99";;
+                DEBUG="-g -m32 -I /home/y/include -Wall -D_REENTRANT -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
+                CFLAGS="-g -O20 -m32 -I /home/y/include -ffast-math -D_REENTRANT -fsigned-char -std=gnu99"
+                PROFILE="-pg -g -O20 -m32 -I /home/y/include -ffast-math -D_REENTRANT -fsigned-char -std=gnu99";;
         sparc-sun-*)
                 DEBUG="-g -Wall -D__NO_MATH_INLINES -fsigned-char -mv8 -std=gnu99"
                 CFLAGS="-O20 -ffast-math -D__NO_MATH_INLINES -fsigned-char -mv8 -std=gnu99"
                 PROFILE="-pg -g -O20 -D__NO_MATH_INLINES -fsigned-char -mv8 -std=gnu99" ;;
         *)
-                DEBUG="-g -Wall -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
-                CFLAGS="-O20 -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
-                PROFILE="-O20 -g -pg -D__NO_MATH_INLINES -fsigned-char -std=gnu99" ;;
+                DEBUG="-g -m32 -I /home/y/include -Wall -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
+                CFLAGS="-g -O20 -m32 -I /home/y/include -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
+                PROFILE="-O20 -g -pg -m32 -I /home/y/include -D__NO_MATH_INLINES -fsigned-char -std=gnu99" ;;
         esac
 fi
 
diff -ruN -x '*m4*' -x 'config.*' -x configure -x 'Makefile.*' -x ltmain.sh old/libdbi-drivers-0.8.3-1/configure.in new/libdbi-drivers-0.8.3-1/configure.in
--- old/libdbi-drivers-0.8.3-1/configure.in	2008-03-03 09:41:41.000000000 -0800
+++ new/libdbi-drivers-0.8.3-1/configure.in	2011-01-05 13:45:48.001227902 -0800
@@ -48,25 +48,25 @@
                 CFLAGS="-xO4 -fast -w -fsimple -native -xcg92"
                 PROFILE="-v -xpg -g -xO4 -fast -native -fsimple -xcg92 -Dsuncc" ;;
         *)
-                DEBUG="-g"
-                CFLAGS="-O"
-                PROFILE="-g -p" ;;
+                DEBUG="-g -m32 -I /home/y/include"
+                CFLAGS="-O -m32 -I /home/y/include"
+                PROFILE="-g -p -m32 -I /home/y/include" ;;
         esac
 else
 
         case $host in
         *-*-linux*)
-                DEBUG="-g -Wall -D_REENTRANT -D__NO_MATH_INLINES -fsigned-char"
-                CFLAGS="-O20 -ffast-math -D_REENTRANT -fsigned-char"
-                PROFILE="-pg -g -O20 -ffast-math -D_REENTRANT -fsigned-char";;
+                DEBUG="-g -m32 -I /home/y/include -Wall -D_REENTRANT -D__NO_MATH_INLINES -fsigned-char"
+                CFLAGS="-g -O20 -m32 -I /home/y/include -ffast-math -D_REENTRANT -fsigned-char"
+                PROFILE="-pg -g -O20 -m32 -I /home/y/include -ffast-math -D_REENTRANT -fsigned-char";;
         sparc-sun-*)
                 DEBUG="-g -Wall -D__NO_MATH_INLINES -fsigned-char -mv8"
                 CFLAGS="-O20 -ffast-math -D__NO_MATH_INLINES -fsigned-char -mv8"
                 PROFILE="-pg -g -O20 -D__NO_MATH_INLINES -fsigned-char -mv8" ;;
         *)
-                DEBUG="-g -Wall -D__NO_MATH_INLINES -fsigned-char"
-                CFLAGS="-O20 -D__NO_MATH_INLINES -fsigned-char"
-                PROFILE="-O20 -g -pg -D__NO_MATH_INLINES -fsigned-char" ;;
+                DEBUG="-g -m32 -I /home/y/include -Wall -D__NO_MATH_INLINES -fsigned-char"
+                CFLAGS="-g -O20 -m32 -I /home/y/include -D__NO_MATH_INLINES -fsigned-char"
+                PROFILE="-O20 -g -pg -m32 -I /home/y/include -D__NO_MATH_INLINES -fsigned-char" ;;
         esac
 fi
 

[libdbi-users] Broken Debug Behavior in libdbi

[Matthew H. <mh...@mh...>]

I am resending because libdbi-* blocks non-subscriber posts.

On Tue, Jan 04, 2011 at 06:20:11PM -0800, Matthew Hall wrote:
Hello,

Sorry about the cross-post, this came up in the interaction of two 
different pieces of software used together.

libdbi's configure.in and other files contain some broken code which is 
preventing me from debugging a segfault in dbi_initialize when this 
function is called by syslog-ng.

1) The code is stripping the libraries of debug symbols when 'make 
install' is called which violates the GNU coding standards.

To fix this, I want to disable library stripping that it is doing which 
is not in line with the standards and would appreciate advice on how to 
do this.

2) I could work around it using the make debug target, but then the 
could would ignore my CFLAGS which are:

-m32 -g -I /home/y/include

Ignoring these causes it to try to generate a 64-bit library which won't 
be compatible with the syslog-ng binary I want to build.

To fix this, I want to make sure that the DEBUG target is not ignoring 
or overriding the 'upstream' user flags, but merely adding to them, so 
that my '-m32' and my '-I' are not lost, which causes build breakage for 
me, and my '-g' gets duplicated.

3) To work around these problems for now I'm going to patch configure.in 
and rerun autoconf unless there is an alternative proposal.

Thanks,
Matthew Hall.

Here is the problematic flag setup being done in configure.in:

if test -z "$GCC"; then
        case $host in
        *-*-irix*)
                if test -z "$CC"; then
                        CC=cc
                fi
                DEBUG="-g -signed"
                CFLAGS="-O2 -w -signed"
                PROFILE="-p -g3 -O2 -signed" ;;
        sparc-sun-solaris*)
                DEBUG="-v -g"
                CFLAGS="-xO4 -fast -w -fsimple -native -xcg92"
                PROFILE="-v -xpg -g -xO4 -fast -native -fsimple -xcg92 -Dsuncc" ;;
        *)
                DEBUG="-g"
                CFLAGS="-O"
                PROFILE="-g -p" ;;
        esac
else

        case $host in
        *-*-linux*)
                DEBUG="-g -Wall -D_REENTRANT -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
                CFLAGS="-O20 -ffast-math -D_REENTRANT -fsigned-char -std=gnu99"
                PROFILE="-pg -g -O20 -ffast-math -D_REENTRANT -fsigned-char -std=gnu99";;
        sparc-sun-*)
                DEBUG="-g -Wall -D__NO_MATH_INLINES -fsigned-char -mv8 -std=gnu99"
                CFLAGS="-O20 -ffast-math -D__NO_MATH_INLINES -fsigned-char -mv8 -std=gnu99"
                PROFILE="-pg -g -O20 -D__NO_MATH_INLINES -fsigned-char -mv8 -std=gnu99" ;;
        *)
                DEBUG="-g -Wall -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
                CFLAGS="-O20 -D__NO_MATH_INLINES -fsigned-char -std=gnu99"
                PROFILE="-O20 -g -pg -D__NO_MATH_INLINES -fsigned-char -std=gnu99" ;;
        esac
fi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Re: [libdbi-users] PATCH: sqlite3 parser bugs

[Vikram A. <noe...@gm...>]

On 12/25/2010 08:34 PM, Vikram Ambrose wrote:
> The first issue I had was that the parser was unable to find Tables in 
> the SQL statement. This was because the parser assumed that the only 
> possible white space around keywords was ' ' or 0x20;
>
> [...]
>
> The second bug was that the parser would always read past the first 
> "endword" if there was a JOIN just before it. Resulting in tables 
> being added to the list that don't exist. (eg. Artist.name, not a 
> table but a field). Thus in order to fix this you need to check for 
> endwords right away if you're parsing through JOINs.
>
>

And here is an update to the patch for a third bug.

The third bug is where the driver is unable to find functions in the SQL 
query.

[...]
if ( function_flag == 1 ) {
    free(statement_copy);
    strcpy(curr_field,itemstore);
[...]

"itemstore" is a pointer into "statement_copy", and as such you cannot 
free it before the strcpy().

The patch moves the free() after the strcpy() and the parser is now able 
to correctly identify function return types.

Again, 380 tests pass, 0 fail.

By the way, Merry Christmas and a Happy New Year to everyone.


Vikram.

[libdbi-users] PATCH: sqlite3 parser bugs

[Vikram A. <noe...@gm...>]

I'm working on a patch for the sqlite3 driver to resolve some of the 
issues I've been having.

The first issue I had was that the parser was unable to find Tables in 
the SQL statement. This was because the parser assumed that the only 
possible white space around keywords was ' ' or 0x20;

eg. strstr(statement," FROM ");

If you had SQL formatted like;

SELECT foo\nFROM bar;

or

SELECT foo\n\tFROM bar;

The parser would fail to find 'bar' because there isn't a ' ' character 
before FROM. This was a problem for all keywords. I think using strtok 
would have been a better approach than going char by char. But since the 
parser works in general, I didn't want to rewrite it all.

The second bug was that the parser would always read past the first 
"endword" if there was a JOIN just before it. Resulting in tables being 
added to the list that don't exist. (eg. Artist.name, not a table but a 
field). Thus in order to fix this you need to check for endwords right 
away if you're parsing through JOINs.

I'm still having all sorts of issues with the sqlite3 driver but these 
two things have at least got my application running :)

I also removed some magic references to nottables[] and added some 
comments to denote the end of large statements.

I ran make check with the patch and 380 tests passed, 0 failed.

Let me know what you think of the patch.

On a side note, write performance to sqlite3 db is incredibly poor (>1s 
for each INSERT regardless of content or no. of fields). But read 
performance is great. Any tips on how to tune sqlite3?


Vikram.

[libdbi-users] sqlite3 multiple tables

[Vikram A. <noe...@gm...>]

Seem to be having a small problem with the sqlite3 driver.

Scenario is 3 tables.

1. Artist (id,name)
2. Album (id,VA,name)
3. Album_Artist (artist_id,album_id)

(VA = is a flag to denote Various Artists type Albums)

Trying to select a particular Artist from a list VA=0 Albums.

SELECT Artist.id,Artist.name
    FROM Artist
    JOIN Album_Artist ON Album_Artist.artist_id = Artist.id
    JOIN Album ON Album.id = Album_Artist.album_id
WHERE Album.VA=0 AND Artist.id=54666831121847624
GROUP BY Artist.name,Artist.id

Now the query works fine directly on the sqlite3 console. But when I 
pass it through libdbi I get a message thrown on the console from the 
driver.

no tables in statement !
no tables in statement !


How do I write that query in a manner compatible with libdbi?


Vikram.

Re: [libdbi-users] sqlite and binary

[Vikram A. <noe...@gm...>]

On 12/23/2010 02:35 PM, mar...@mh... wrote:
> Vikram Ambrose writes:
>   >  I can't seem to get BLOBs to work with SQLite.
>   >
>   >  Here is some test code.
>   >  http://en.pastebin.ca/2026862
>   >
>   >  Output on my machine is;
>   >
>   >  dbi_conn_connect: rv = 0:
>   >  data = ERROR
>   >
>   >  What am I doing wrong?
>
> Could you test sqlite3 as well just to see whether it is a problem of your box
> or indeed a problem of the driver.
>

Interestingly enough the test code works fine under sqlite3. Looks like 
a bug in the sqlite(2) driver so far.

> Did you run make check in libdbi-drivers after building and installing
> the drivers? The test program includes checks for blob handling, so if
> there is a problem in the driver it should report errors. Runs fine here.
>

make check gives me an error. Missing dependency probably?

test_dbi.c:33: fatal error: cgreen/cgreen.h: No such file or directory


> Also, could you send a few specs of your OS and libdbi/libdbi-drivers
> versions?


Using CVS versions of both. Check'd out yesterday.

The strlen() was used because I knew what the input data was.

On a side note, in general, I often use BLOB instead of VARCHAR/STRING 
for clear text strings as its much easier to deal with character 
encoding/locale at the application level than messing around with it at 
the DB level.

I suppose I'll just switchover to sqlite3. Doesn't make a difference to 
me really. Just needed a database that could be used for a test run 
without the user requiring administrative privileges.

thanks,


Vikram.

[libdbi-users] sqlite and binary

[<mar...@mh...>]

Vikram Ambrose writes:
 > I can't seem to get BLOBs to work with SQLite.
 > 
 > Here is some test code.
 > http://en.pastebin.ca/2026862
 > 
 > Output on my machine is;
 > 
 > dbi_conn_connect: rv = 0:
 > data = ERROR
 > 
 > What am I doing wrong?

Probably nothing, at least not in your own code. Your test code prints
the following on my box:

$ ./sqlite_binary_test
dbi_conn_connect: rv = 0: 
dbi_conn_query: 
data = close your eyes, this is binary

All I changed is the driver. I've used sqlite3 as I currently don't
have sqlite installed. However, as far as I can see there is no
difference between those drivers in terms of blob handling. Could you
test sqlite3 as well just to see whether it is a problem of your box
or indeed a problem of the driver.

Did you run make check in libdbi-drivers after building and installing
the drivers? The test program includes checks for blob handling, so if
there is a problem in the driver it should report errors. Runs fine here.

Also, could you send a few specs of your OS and libdbi/libdbi-drivers
versions?

regards,
Markus

P.S. your code contains a strlen() call on binary data. I assume you
are aware that this wouldn't work with arbitrary binary data.

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[libdbi-users] sqlite and binary

[Vikram A. <noe...@gm...>]

I can't seem to get BLOBs to work with SQLite.

Here is some test code.
http://en.pastebin.ca/2026862

Output on my machine is;

dbi_conn_connect: rv = 0:
data = ERROR

What am I doing wrong?



Vikram.

[libdbi-users] maintenance release 0.8.4 available

[Markus H. <mar...@mh...>]

Hi all,

after a longer hiatus there is a new libdbi release available for
download. This is merely a maintenance release which was requested by
the Debian and Ubuntu packagers (see Bug#592447 in the Debian bug
tracker). There are no end-user-visible changes compared to the
previous release, so if you have 0.8.3 installed there are no benefits
of installing the new version.

The following changes were implemented:

- bumped the package version number to 0.8.4

- updated some autotools-related things in configure.in, autogen.sh,
  and Makefile.am (recent versions need AC_CONFIG_MACRO_DIR set
  properly, otherwise I wouldn't be able to bootstrap the build on my
  FreeBSD development box)

- LIB_CURRENT=1, LIB_REVISION=0, LIB_AGE=0

- backported the --disable-docs behaviour fix in configure from HEAD

Bumping up LIB_CURRENT was necessary due to backwards-incompatible
changes in constants. We had planned to handle the library versioning
thing appropriately starting with 0.9.0, but the Debian folks were
eager to upgrade their (even more outdated) 0.8.2 packages to
0.8.3. This would have screwed up dozens of their packages unless the
so name issue was fixed.

Our greatly enhanced, all new and shiny 0.9.0 will be shipped real
soon now (TM). However, the Debian folks have suggested a bunch of
upstream patches which I'm inclined to include in the next release, so
don't hang on the edges of your seats.

regards,
Markus


-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [libdbi-users] Timeout / Reconnect

[Vikram A. <noe...@gm...>]

On 07/15/2010 04:11 PM, Markus Hoenicka wrote:
>   I assume that the server going away is a problem of all database engines, except maybe the
> embedded ones. Adding some code to the MySQL driver to fix your
> original problem thus might be the wrong solution. Don't you
> experience the same problems with other DB engines? In that case, we'd
> have to fix libdbi, rather than a driver.
>
>    

I'm not sure Markus. I'm almost exclusively been using Mysql for a while 
now. In either case, I doubt a gone-awayreconnect/retransmit fix would 
be too intrusive to libdbi or the driver.

Re: [libdbi-users] Timeout / Reconnect

[Markus H. <mar...@mh...>]

Vikram Ambrose writes:
 > There is indeed enough information to find this scenario. Here is a very 
 > simple callback I use at the moment:
 > 
 > void DB_error_callback(dbi_conn conn, void *udata){
 >      int rv;
 >      const char *errormsg = NULL;
 > 
 >      rv = dbi_conn_error(DB_conn,&errormsg);
 >      if(strstr(errormsg,"gone away")){
 >          rv = dbi_conn_connect(DB_conn);
 >          if(rv){
 >              printf("ERROR: DB_error_callback: Tried to reconnect - 
 > failed\n");
 >          }else {
 >              printf("ERROR: DB_error_callback: Reconnected\n");
 >          }
 >      }
 > }
 > 

Hi,

I just found some time to fiddle with this. As it seems, your solution
is pretty much focused on MySQL. However, I assume that the server
going away is a problem of all database engines, except maybe the
embedded ones. Adding some code to the MySQL driver to fix your
original problem thus might be the wrong solution. Don't you
experience the same problems with other DB engines? In that case, we'd
have to fix libdbi, rather than a driver.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38