Re: [Perfparse-devel] libdbi

[Ben C. <Be...@cl...>]

Sounds good.

Although I am not overjoyed at having to suggest to users that they need 
to install another program first, it will be a once only, and it sounds 
like the benefits are greater than the problems.  I would however like 
to check that libdbi installs cleanly and does not add major support 
problems.

Another thing I want to look seriously at is the cursors in libdbi.

PerfParse uses two cursors, both of which need supporting.

The standard cursor which returns an entire table in memory.  This is 
fast when the table is small, and allows us to get the row count, which 
is used many times in the code.

The read-forward cursor which replies a row at a time.  This is 
essential for the graph drawing.  Loading a table with 100,000 rows is 
slow and consumes way too much memory.

The respective calls in the MySQL APR are:

query_result = mysql_store_result(&db_mysql);
query_result = mysql_use_result(&db_mysql);

I know that PostgreSQL only supports the former, which may be the case 
with other DBMS as well.

In the same way, binary or ASCII cursors will need supporting, where 
available from the DBMS.

If this is not the case, then a simple API for doing this our selves is 
not hard to write.  A set of .c files for each DB, each of which have a 
common .h file, and are linked, or not, depending on what configure 
option has been passed.

I emplisise how simple the API needs to be.

All our data is text, no attempt at correct local storage is done.  We 
only use one table at a time.  We operate a 'no error' policy.  Any SQL 
error will cause the program to exit.  Error interperation is therefore 
not needed.  All SQL is simple text without the need for variables or 
clever escapement.  We use two simple calls only to execute the SQL.

I say this only as I have used API's in .NET, Java, ODBC, ADO, and many 
UNIX MySQL API's.  As well as at least three I have written my self.

Most are vastly over complex, hard to use, hard to understand, and 
ultimately far more complex than direct calls.  I am not saying this is 
the case for libdbi, it may be the exception.  But lets not jump into 
bed with a monster. :)

For what it's worth, Ben.





Yves Mettier wrote:

>>Ben and Yves,
>>
>>Thanks for your warm responses.
> 
> 
> :)
> 
> 
>>I agree that the lack of users listed on the libdbi website doesn't
>>look particularly promising however, I think it is probably the correct
>>direction to go, assuming that we don't run into performance issues
>>that require customized database code. A company that I work for, uses
>>the perl package DBI for all of our web applications that interface
>>with databases. I don't believe that DBI for perl and libdbi are
>>related except that they serve the same purpose for their respective
>>languages.
> 
> 
> This is the same reason I think we should give a try :)
> 
> But there is one more check that I must do before we give it a try: It has to compile on
> Solaris. Otherwise, why try to use it to port on other databases if you cannot port it
> to other OSes ?
> For your information, I work on Solaris, and we have already had some people testing
> perfparse on FreeBSD (with no success/failure feedback)
> 
> 
>>Based on an initial comparison of the MySQL code in PerfParse 0.10 and
>>the MySQL dbd module for libdbi, It doesn't look particularly difficult
>>to change from the current MySQL specific calls to libdbi calls. Has
>>the database interface code been changed significantly for 0.11?
> 
> 
> New numerotation: now 0.0.11 :)
> 
> The database interface has not changed, but the table definitions will change a lot.
> Ben will send us a new mysql_create.sql file, then I work on all the tools to make them
> work with it.
> We still have some points to discuss about it, but it's nearly done. Will probably be
> ready tomorrow (thursday).
> 
> 
>>I would like to propose the following plan of action:
>>1. Produce a patch for 0.11 which enables a configuration option to use
>>libdbi in stead of direct MySQL code. (MySQL code is still default)
>>2. Role the patch into the 0.12 release
>>3. Down the road (0.13 or later), make libdbi the default and MySQL an
>>option.
>>4. Much farther down the road, we can remove the MySQL option all
>>together (or not).
> 
> 
> I have some other idea, because we will be 3 (including you) to work on the whole code.
> And both Ben and me have no access to CVS (firewall...)
> 
> 1/
> I suggest that someone work on perfparse-install-db first. I can do it fast because I
> won't use libdbi.
> Then you take my patch to perfparse-install-db and move it to libdbi.
> 
> 2/
> I work on perfparse-purge-db (without libdbi) while you can already work on perfgraph
> and perfchart (adaptation to the new table definitions, and use libdbi).
> 
> 3/
> Somebody adapt perfparse (without libdbi)
> 
> Ben releases 0.0.12 (with libdbi working with perfparse-install-db, perfgraph and
> perfchart).
> 
> 
> 0.0.13 will use libdbi everywhere except in perfparse.
> 
> After that, depending on how we work, in 0.0.14 or after, we can have an optionnal
> support of libdbi in perfparse, in order to see its performances.
> 
> 
> 
> Some functions use mysql API. Please leave them as is. Write some other functions that
> do the same so that we can still use mysql API. Some functions are shared between all
> the perfparse tools !
> I think we will not need any #ifdef, and it's better that way. But I may be wrong :)
> 
> 
>>All of this is dependent on getting reasonable performance using
>>libdbi. If libdbi is unusable, I will scrap it in favor of writing code
>>for each database. If it is usable but not as good as MySQL directly,
>>we just might not ever make it to steps 3 and 4 above.
>>
>>Let me know how this sounds and I am anxiously awaiting the release of
>>0.11.
> 
> 
> Released already, at 16h today (well, is it really 16h ? All files are released at 16h :)
> 
> Yves
> 
> 
>>Chris
>>
>>
>>
>>-------------------------------------------------------
>>This SF.Net email sponsored by Black Hat Briefings & Training.
>>Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
>>digital self defense, top technical experts, no vendor pitches,
>>unmatched networking opportunities. Visit www.blackhat.com
>>_______________________________________________
>>Perfparse-devel mailing list
>>Per...@li...
>>https://lists.sourceforge.net/lists/listinfo/perfparse-devel
>>
>>
> 
> 
>