Menu
▾
▴
gusdev-gusdev — Topics concerning GUS development
You can subscribe to this list here.
| 2002 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(11) |
Jul
(34) |
Aug
(14) |
Sep
(10) |
Oct
(10) |
Nov
(11) |
Dec
(6) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2003 |
Jan
(56) |
Feb
(76) |
Mar
(68) |
Apr
(11) |
May
(97) |
Jun
(16) |
Jul
(29) |
Aug
(35) |
Sep
(18) |
Oct
(32) |
Nov
(23) |
Dec
(77) |
| 2004 |
Jan
(52) |
Feb
(44) |
Mar
(55) |
Apr
(38) |
May
(106) |
Jun
(82) |
Jul
(76) |
Aug
(47) |
Sep
(36) |
Oct
(56) |
Nov
(46) |
Dec
(61) |
| 2005 |
Jan
(52) |
Feb
(118) |
Mar
(41) |
Apr
(40) |
May
(35) |
Jun
(99) |
Jul
(84) |
Aug
(104) |
Sep
(53) |
Oct
(107) |
Nov
(68) |
Dec
(30) |
| 2006 |
Jan
(19) |
Feb
(27) |
Mar
(24) |
Apr
(9) |
May
(22) |
Jun
(11) |
Jul
(34) |
Aug
(8) |
Sep
(15) |
Oct
(55) |
Nov
(16) |
Dec
(2) |
| 2007 |
Jan
(12) |
Feb
(4) |
Mar
(8) |
Apr
|
May
(19) |
Jun
(3) |
Jul
(1) |
Aug
(6) |
Sep
(12) |
Oct
(3) |
Nov
|
Dec
|
| 2008 |
Jan
(4) |
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
(21) |
| 2009 |
Jan
|
Feb
(2) |
Mar
(1) |
Apr
|
May
(1) |
Jun
(8) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2010 |
Jan
|
Feb
(1) |
Mar
(4) |
Apr
(3) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
(4) |
May
(19) |
Jun
(14) |
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2012 |
Jan
|
Feb
|
Mar
(22) |
Apr
(12) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2013 |
Jan
(2) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(3) |
Jun
|
Jul
|
Aug
(2) |
Sep
|
Oct
|
Nov
|
Dec
(1) |
| 2016 |
Jan
(1) |
Feb
(1) |
Mar
|
Apr
(1) |
May
|
Jun
(2) |
Jul
(1) |
Aug
|
Sep
|
Oct
(1) |
Nov
(1) |
Dec
|
| 2017 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Steve F. <st...@pc...> - 2003-12-06 20:58:32
|
chetna- a couple of possibilities. first, are you sure that the plugin hasn't been mistakenly changed? if you aren't sure, get a fresh copy of the file from the cvs site. it is still on version 1.10. second, is it possible that your md5 program has changed? if neither of those has changed, then somehow your db row for that plugin is wrong. terry's suggestion will work as a quick fix in that case. (thanks terry) steve Terry Clark wrote: >Chetna, you might change the cvs version >in LoadPfam.pm from > cvsRevision => '$Revision: 1.10 >to something like > cvsRevision => '$Revision: 1.10001 >and run ga +update. > >Terry > > >On 0, Chetna Warade <wa...@ya...> wrote: > > >>Hello all, >> >>I am getting an md5 checksum error. LoadPfam has not >>been changed fromlast 6 months. I wonder how to fix >>this problem. >> >>heres the snapshot: >> >>ga GUS::Common::Plugin::LoadPfam >>--flat_file=./Pfam-A.full.gz --release='11.0' --commit >>Reading properties from >>/var/local/gus_home/config/GUS-PluginMgr.prop >>Reading properties from /home/chetna/.gus.properties >> >>USER ERROR: The md5 checksum of >>GUS::Common::Plugin::LoadPfam's executable file (cvs >>revision 1.10) doesn't match the md5 checksum in the >>database for that plugin and revision. IE, the plugin >>has been changed but not commited and updated. >>Please: >> - cvs commit the plugin file >> - use the build system to install >>it >> - run 'ga +update >>GUS::Common::Plugin::LoadPfam --commit' >>Aborting >>Issuing rollback() for database handle being DESTROY'd >>without explicit disconnect(). >> >>Thanks, >>Chetna >> >>__________________________________ >>Do you Yahoo!? >>New Yahoo! Photos - easier uploading and sharing. >>http://photos.yahoo.com/ >> >> >>------------------------------------------------------- >>This SF.net email is sponsored by: SF.net Giveback Program. >>Does SourceForge.net help you be more productive? Does it >>help you create better code? SHARE THE LOVE, and help us help >>YOU! Click Here: http://sourceforge.net/donate/ >>_______________________________________________ >>Gusdev-gusdev mailing list >>Gus...@li... >>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev >> >> > > >------------------------------------------------------- >This SF.net email is sponsored by: SF.net Giveback Program. >Does SourceForge.net help you be more productive? Does it >help you create better code? SHARE THE LOVE, and help us help >YOU! Click Here: http://sourceforge.net/donate/ >_______________________________________________ >Gusdev-gusdev mailing list >Gus...@li... >https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev > > |
|
From: Terry C. <tw...@cs...> - 2003-12-06 20:40:18
|
Chetna, you might change the cvs version
in LoadPfam.pm from
cvsRevision => '$Revision: 1.10
to something like
cvsRevision => '$Revision: 1.10001
and run ga +update.
Terry
On 0, Chetna Warade <wa...@ya...> wrote:
>
> Hello all,
>
> I am getting an md5 checksum error. LoadPfam has not
> been changed fromlast 6 months. I wonder how to fix
> this problem.
>
> heres the snapshot:
>
> ga GUS::Common::Plugin::LoadPfam
> --flat_file=./Pfam-A.full.gz --release='11.0' --commit
> Reading properties from
> /var/local/gus_home/config/GUS-PluginMgr.prop
> Reading properties from /home/chetna/.gus.properties
>
> USER ERROR: The md5 checksum of
> GUS::Common::Plugin::LoadPfam's executable file (cvs
> revision 1.10) doesn't match the md5 checksum in the
> database for that plugin and revision. IE, the plugin
> has been changed but not commited and updated.
> Please:
> - cvs commit the plugin file
> - use the build system to install
> it
> - run 'ga +update
> GUS::Common::Plugin::LoadPfam --commit'
> Aborting
> Issuing rollback() for database handle being DESTROY'd
> without explicit disconnect().
>
> Thanks,
> Chetna
>
> __________________________________
> Do you Yahoo!?
> New Yahoo! Photos - easier uploading and sharing.
> http://photos.yahoo.com/
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> Does SourceForge.net help you be more productive? Does it
> help you create better code? SHARE THE LOVE, and help us help
> YOU! Click Here: http://sourceforge.net/donate/
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
|
|
From: Chetna W. <wa...@ya...> - 2003-12-06 05:22:57
|
Hello all,
I am getting an md5 checksum error. LoadPfam has not
been changed fromlast 6 months. I wonder how to fix
this problem.
heres the snapshot:
ga GUS::Common::Plugin::LoadPfam
--flat_file=./Pfam-A.full.gz --release='11.0' --commit
Reading properties from
/var/local/gus_home/config/GUS-PluginMgr.prop
Reading properties from /home/chetna/.gus.properties
USER ERROR: The md5 checksum of
GUS::Common::Plugin::LoadPfam's executable file (cvs
revision 1.10) doesn't match the md5 checksum in the
database for that plugin and revision. IE, the plugin
has been changed but not commited and updated.
Please:
- cvs commit the plugin file
- use the build system to install
it
- run 'ga +update
GUS::Common::Plugin::LoadPfam --commit'
Aborting
Issuing rollback() for database handle being DESTROY'd
without explicit disconnect().
Thanks,
Chetna
__________________________________
Do you Yahoo!?
New Yahoo! Photos - easier uploading and sharing.
http://photos.yahoo.com/
|
|
From: Deborah F. P. <pi...@pc...> - 2003-12-05 18:04:03
|
Hi Chetna, The external_db_rel_id attribute holds the sres.externaldatabaserelease id for each of the source databases that are found in the defline of nr protein db records. I have modified the plugin and submitted it to cvs because we are now maintaining two versions of the ncbi nr protein database. If you update the new command line should be similar to the following: --temp_login $temp_login --sourceDB $sourceDB --temp_password $temp_password --dbi_str $dbi_str --gitax $gitax --nrdb $nrdb --extDbRelId $nrdbReleaseId --maketemp --plugin --delete $temp_login = pinney $temp_password = xxxxxx $sourceDB = 7475:gb,7476:emb,7477:dbj,7478:pir,7479:prf,7480:sp,7481:pdb,7514:pat, 7515:bbs,7516:gnl,7474:ref,7517:lcl,7482:genpept,7483:tpe $dbi_str = dbi:Oracle:host=cbilbld.pcbi.upenn.edu;sid=cbilbld $nrdbReleaseId = 7484 etc. The previous version took the name of the source database and retrieved the external_database_release_id while this version is given the id. I think this is better because you don't have to run the risk of mistyping the name and allows you to maintain more than one version of nrdb. It still requires that you enter a row into sres.externaldatabase and sres.externaldatabaserelease for each of the source db in the NCBI files (see the README file at NCBI). (We have a couple extra here - if you can query our db, you can get their descriptions). The ids can be permanent so you don't have to come up with the command line and new ids every time. You also need to have a separate NRDB external_database_release_id for NRDB on the whole and this will be put into dots.externalaasequence. We also don't change this id, rather we update information in the sres.externaldatabaserelease table row. Debbie On Fri, 5 Dec 2003, Chetna Warade wrote: > Hello, > > I tried with --maketemp and I am getting: > > DBD::Oracle::st execute failed: ORA-01400: cannot > insert NULL into > ("CHETNA"."NRDBTEMP"."EXTERNAL_DB_REL_ID") (DBD ERROR: > OCIStmtExecute) at > /var/local/gus_home/lib/perl/GUS/Common/Plugin/LoadNRDB.pm > line 270, <NRDB> line 156693. > > Let me know what is null? > > Thanks > Chetna > > --- "Deborah F. Pinney" <pi...@pc...> wrote: > > Hi Chetna, > > > > The plugin was written so that it could be run in > > three parts: 1) making > > the temp table, 2) inserting into the NRDBEntry and > > ExternalAASequences > > tables, 3) deleting obsolete rows in those tables. I > > looked back at your > > previous e-mail and it looks as if you have not > > included the option to > > create and fill the temp table, --maketemp. The > > plugin should create the > > table if that is included on the command line. You > > can see the table > > schema by looking into the code, sub makeTempTable. > > I'm hoping to rewrite > > this plugin as it is complicated and could be put in > > a better form. > > > > > > > > Debbie > > > > > __________________________________ > Do you Yahoo!? > Free Pop-Up Blocker - Get it now > http://companion.yahoo.com/ > |
|
From: Chetna W. <wa...@ya...> - 2003-12-05 17:32:10
|
Hello,
I tried with --maketemp and I am getting:
DBD::Oracle::st execute failed: ORA-01400: cannot
insert NULL into
("CHETNA"."NRDBTEMP"."EXTERNAL_DB_REL_ID") (DBD ERROR:
OCIStmtExecute) at
/var/local/gus_home/lib/perl/GUS/Common/Plugin/LoadNRDB.pm
line 270, <NRDB> line 156693.
Let me know what is null?
Thanks
Chetna
--- "Deborah F. Pinney" <pi...@pc...> wrote:
> Hi Chetna,
>
> The plugin was written so that it could be run in
> three parts: 1) making
> the temp table, 2) inserting into the NRDBEntry and
> ExternalAASequences
> tables, 3) deleting obsolete rows in those tables. I
> looked back at your
> previous e-mail and it looks as if you have not
> included the option to
> create and fill the temp table, --maketemp. The
> plugin should create the
> table if that is included on the command line. You
> can see the table
> schema by looking into the code, sub makeTempTable.
> I'm hoping to rewrite
> this plugin as it is complicated and could be put in
> a better form.
>
>
>
> Debbie
>
__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/
|
|
From: Steve F. <st...@pc...> - 2003-12-05 17:05:40
|
All CBIL sites use gus 3.0. The previous schema is not in use here. AllGenes, EPConDB and PlasmoDb are using the current WDK-Classic. ApiEstDB is about to be upgraded. Steve Adrian Roy Tivey wrote: >Hi, > >Just chatting to Angel this week, we didn't realise that any of the public sites >had been ported to GUS3. A couple of questions: > >a) Which of the current public CBIL sites run GUS3? >b) Which of these has the latest version of the classic WDK? (Not including >EPConDB 3.2 which is the very latest version as I understand it) > >Adrian > > > >------------------------------------------------------- >This SF.net email is sponsored by: SF.net Giveback Program. >Does SourceForge.net help you be more productive? Does it >help you create better code? SHARE THE LOVE, and help us help >YOU! Click Here: http://sourceforge.net/donate/ >_______________________________________________ >Gusdev-gusdev mailing list >Gus...@li... >https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev > > |
|
From: Deborah F. P. <pi...@pc...> - 2003-12-05 17:00:54
|
Hi Chetna,
The plugin was written so that it could be run in three parts: 1) making
the temp table, 2) inserting into the NRDBEntry and ExternalAASequences
tables, 3) deleting obsolete rows in those tables. I looked back at your
previous e-mail and it looks as if you have not included the option to
create and fill the temp table, --maketemp. The plugin should create the
table if that is included on the command line. You can see the table
schema by looking into the code, sub makeTempTable. I'm hoping to rewrite
this plugin as it is complicated and could be put in a better form.
Debbie
|
|
From: Adrian R. T. <ar...@sa...> - 2003-12-05 16:58:24
|
Hi, Just chatting to Angel this week, we didn't realise that any of the public sites had been ported to GUS3. A couple of questions: a) Which of the current public CBIL sites run GUS3? b) Which of these has the latest version of the classic WDK? (Not including EPConDB 3.2 which is the very latest version as I understand it) Adrian |
|
From: Chetna W. <wa...@ya...> - 2003-12-05 16:44:26
|
Hello Steve and Debbie, Heres the snapshot. Do I need to create the temp table NRDBTemp (whats the schmema?) for the user account. Are there any other tables and what is their schema? It will be great if I could get the schema scripts. There are 12 entries in the database hash Fri Dec 5 11:27:00 2003 There are 3233042 gi to taxon_id pairs DBD::Oracle::db prepare failed: ORA-00942: table or view does not exist (DBD ERROR: OCIStmtExecute/Describe) at /var/local/gus_home/lib/perl/GUS/ObjRelP/DbiDbHandle.pm line 78. Prepare FAILED: ORA-00942: table or view does not exist (DBD ERROR: OCIStmtExecute/Describe) sql_cmd: select max(set_num) from chetna.NRDBTemp Can't call method "execute" without a package or object reference at /var/local/gus_home/lib/perl/GUS/ObjRelP/DbiDbHandle.pm line 79. Thanks Chetna --- Steve Fischer <st...@pc...> wrote: > Chetna- > > the latest version of > $PROJECT_HOME/GUS/ObjRelP/lib/perl/DbiDbHandle.pm > has a fix for that error message. it now shows the > problematic sql. > > can you please get it from the cvs site and then do > a build. > > (before you do, save your current DbiDbHandle.pm > just in case) > > steve > > Chetna Warade wrote: > > >Hello, > > > >I am getting following error while I am trying to > load > >NRDB into GUS. Which table is it complaining for, I > >tried with --debug or --verbose but ga didnt show > any > >useful error message. I will appreciate any help. > > > > > >ga GUS::Common::Plugin::LoadNRDB --temp_login > chetna > >--sourceDB 'GENBANK (NRDB):gb,EMBL DATA LIBRARY > >(NRDB):emb,DDBJ (NRDB):dbj,NBRF PIR > (NRDB):pir,PROTEIN > >RESEARCH FOUNDATION (NRDB):prf,SWISS-PROT > >(NRDB):sp,BROOKHAVEN PROTEIN DATA BANK > >(NRDB):pdb,PATENTS (NRDB):pat,GENINFO BACKBONE ID > >(NRDB):bbs,GENERAL DATABASE IDENTIFIER > (NRDB):gnl,NCBI > >REFERENCE SEQUENCE (NRDB):ref,LOCAL SEQUENCE > >IDENTIFIER:lcl' --temp_password chetna123 --dbi_str > >'dbi:Oracle:host=kiwi.rcr.uga.edu;sid=CTEGD' > --gitax > >/home/chetna/gus-app/ncbi-taxonomy/gi_taxid_prot.dmp > >--nrdb > >/opt/app/oracle/db_storage/data/downloadNRDB/downloadNRDB/nr.Z > >--extDbRelId 15 --plugin --delete > >Wed Dec 3 17:09:57 2003 **COMMIT TURNED > OFF** > > > >There are 12 entries in the database hash > >Wed Dec 3 17:10:30 2003 There are 3233042 > gi > >to taxon_id pairs > > > >DBD::Oracle::db prepare failed: ORA-00942: table or > >view does not exist (DBD ERROR: > >OCIStmtExecute/Describe) at > >/var/local/gus_home/lib/perl/GUS/ObjRelP/DbiDbHandle.pm > >line 78. > >prepareAndExecute FAILED: > >GUS::ObjRelP::DbiDbHandle=HASH(0x86c9d48)->errstr > > > >Thanks, > >Chetna > > > > > > > >__________________________________ > >Do you Yahoo!? > >Free Pop-Up Blocker - Get it now > >http://companion.yahoo.com/ > > > > > >------------------------------------------------------- > >This SF.net email is sponsored by: SF.net Giveback > Program. > >Does SourceForge.net help you be more productive? > Does it > >help you create better code? SHARE THE LOVE, and > help us help > >YOU! Click Here: http://sourceforge.net/donate/ > >_______________________________________________ > >Gusdev-gusdev mailing list > >Gus...@li... > >https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev > > > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback > Program. > Does SourceForge.net help you be more productive? > Does it > help you create better code? SHARE THE LOVE, and > help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Gusdev-gusdev mailing list > Gus...@li... > https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev __________________________________ Do you Yahoo!? Free Pop-Up Blocker - Get it now http://companion.yahoo.com/ |
|
From: Angel P. <an...@pc...> - 2003-12-04 11:41:15
|
On Dec 4, 2003, at 3:51 AM, Steve Fischer wrote: > One relatively-easy-to-implement possibility for the WDK would be a > mini-testing framework to *syntax check* the queries in the > servlet-config files. It could be given a handle on all the queries, > along with parameter values (that part is tricky because the db might > change obsoleting the values). It would then simply check to see if > the queries run, and return any values. IE, don't try to get to > regression test levels of comparing the result with known answers: > that requires a static test dataset. > This is more of what I was thinking of, thanks for hashing out my late-night rambling email. I see what you mean about the test dataset instance, but in reality, most of the queries we write for any given website is built from previous knowledge of what you want to get out of the query. Hence, if you are writing queries, you should be at least a bit familiar with the expected results. One last point, in most cases a "null" return on a DB query is a perfectly valid result (and sometimes desirable) . Angel > steve > > Angel Pizarro wrote: > >> I was hashing some stuff out with Andy today and came up with the >> idea for the WDK and want to bounce it off of the general list. >> >> How about a unit test for websites? If we can have an automatic test >> of the site before install, that would really help out for a) testing >> queries on a non-live site, b) make sure that the content has no dead >> links, b) catch potential bugs in procedural code (e.g. CGI scripts >> or any other non-compiled code) >> >> Any thoughts? >> Angel >> >> >> >> ------------------------------------------------------- >> This SF.net email is sponsored by: SF.net Giveback Program. >> Does SourceForge.net help you be more productive? Does it >> help you create better code? SHARE THE LOVE, and help us help >> YOU! Click Here: http://sourceforge.net/donate/ >> _______________________________________________ >> Gusdev-gusdev mailing list >> Gus...@li... >> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Gusdev-gusdev mailing list > Gus...@li... > https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev |
|
From: Steve F. <st...@pc...> - 2003-12-04 03:43:20
|
One relatively-easy-to-implement possibility for the WDK would be a mini-testing framework to *syntax check* the queries in the servlet-config files. It could be given a handle on all the queries, along with parameter values (that part is tricky because the db might change obsoleting the values). It would then simply check to see if the queries run, and return any values. IE, don't try to get to regression test levels of comparing the result with known answers: that requires a static test dataset. steve Angel Pizarro wrote: > I was hashing some stuff out with Andy today and came up with the idea > for the WDK and want to bounce it off of the general list. > > How about a unit test for websites? If we can have an automatic test > of the site before install, that would really help out for a) testing > queries on a non-live site, b) make sure that the content has no dead > links, b) catch potential bugs in procedural code (e.g. CGI scripts or > any other non-compiled code) > > Any thoughts? > Angel > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Gusdev-gusdev mailing list > Gus...@li... > https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev |
|
From: Angel P. <an...@pc...> - 2003-12-03 23:18:51
|
Just a quick note before I forget to I go to sleep, but I would really love to avoid a mix of JSF and Struts. From the one article I read, Struts was not meant to handle JSF components and needs a lot of code to catch sends/requests to/from JSF. Struts version 2 may change all of this, but it is not even in alpha stage. And I have a feeling that Struts v2 will have little to no code re-use from current Struts. I think either a Struts/Tiles or JSF approach will both be more than we need to do the two things the WDK needs: 1) A web UI 2) webservices or other socket query mechanisms While I also think that component-based Model/View would be very cool, it may be over-engineering the 2 criteria we have for the UI ANgel |
|
From: Angel P. <an...@pc...> - 2003-12-03 23:12:56
|
I was hashing some stuff out with Andy today and came up with the idea for the WDK and want to bounce it off of the general list. How about a unit test for websites? If we can have an automatic test of the site before install, that would really help out for a) testing queries on a non-live site, b) make sure that the content has no dead links, b) catch potential bugs in procedural code (e.g. CGI scripts or any other non-compiled code) Any thoughts? Angel |
|
From: Steve F. <st...@pc...> - 2003-12-03 22:21:10
|
Chetna- the latest version of $PROJECT_HOME/GUS/ObjRelP/lib/perl/DbiDbHandle.pm has a fix for that error message. it now shows the problematic sql. can you please get it from the cvs site and then do a build. (before you do, save your current DbiDbHandle.pm just in case) steve Chetna Warade wrote: >Hello, > >I am getting following error while I am trying to load >NRDB into GUS. Which table is it complaining for, I >tried with --debug or --verbose but ga didnt show any >useful error message. I will appreciate any help. > > >ga GUS::Common::Plugin::LoadNRDB --temp_login chetna >--sourceDB 'GENBANK (NRDB):gb,EMBL DATA LIBRARY >(NRDB):emb,DDBJ (NRDB):dbj,NBRF PIR (NRDB):pir,PROTEIN >RESEARCH FOUNDATION (NRDB):prf,SWISS-PROT >(NRDB):sp,BROOKHAVEN PROTEIN DATA BANK >(NRDB):pdb,PATENTS (NRDB):pat,GENINFO BACKBONE ID >(NRDB):bbs,GENERAL DATABASE IDENTIFIER (NRDB):gnl,NCBI >REFERENCE SEQUENCE (NRDB):ref,LOCAL SEQUENCE >IDENTIFIER:lcl' --temp_password chetna123 --dbi_str >'dbi:Oracle:host=kiwi.rcr.uga.edu;sid=CTEGD' --gitax >/home/chetna/gus-app/ncbi-taxonomy/gi_taxid_prot.dmp >--nrdb >/opt/app/oracle/db_storage/data/downloadNRDB/downloadNRDB/nr.Z >--extDbRelId 15 --plugin --delete >Wed Dec 3 17:09:57 2003 **COMMIT TURNED OFF** > >There are 12 entries in the database hash >Wed Dec 3 17:10:30 2003 There are 3233042 gi >to taxon_id pairs > >DBD::Oracle::db prepare failed: ORA-00942: table or >view does not exist (DBD ERROR: >OCIStmtExecute/Describe) at >/var/local/gus_home/lib/perl/GUS/ObjRelP/DbiDbHandle.pm >line 78. >prepareAndExecute FAILED: >GUS::ObjRelP::DbiDbHandle=HASH(0x86c9d48)->errstr > >Thanks, >Chetna > > > >__________________________________ >Do you Yahoo!? >Free Pop-Up Blocker - Get it now >http://companion.yahoo.com/ > > >------------------------------------------------------- >This SF.net email is sponsored by: SF.net Giveback Program. >Does SourceForge.net help you be more productive? Does it >help you create better code? SHARE THE LOVE, and help us help >YOU! Click Here: http://sourceforge.net/donate/ >_______________________________________________ >Gusdev-gusdev mailing list >Gus...@li... >https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev > > |
|
From: Chetna W. <wa...@ya...> - 2003-12-03 22:16:36
|
Hello, I am getting following error while I am trying to load NRDB into GUS. Which table is it complaining for, I tried with --debug or --verbose but ga didnt show any useful error message. I will appreciate any help. ga GUS::Common::Plugin::LoadNRDB --temp_login chetna --sourceDB 'GENBANK (NRDB):gb,EMBL DATA LIBRARY (NRDB):emb,DDBJ (NRDB):dbj,NBRF PIR (NRDB):pir,PROTEIN RESEARCH FOUNDATION (NRDB):prf,SWISS-PROT (NRDB):sp,BROOKHAVEN PROTEIN DATA BANK (NRDB):pdb,PATENTS (NRDB):pat,GENINFO BACKBONE ID (NRDB):bbs,GENERAL DATABASE IDENTIFIER (NRDB):gnl,NCBI REFERENCE SEQUENCE (NRDB):ref,LOCAL SEQUENCE IDENTIFIER:lcl' --temp_password chetna123 --dbi_str 'dbi:Oracle:host=kiwi.rcr.uga.edu;sid=CTEGD' --gitax /home/chetna/gus-app/ncbi-taxonomy/gi_taxid_prot.dmp --nrdb /opt/app/oracle/db_storage/data/downloadNRDB/downloadNRDB/nr.Z --extDbRelId 15 --plugin --delete Wed Dec 3 17:09:57 2003 **COMMIT TURNED OFF** There are 12 entries in the database hash Wed Dec 3 17:10:30 2003 There are 3233042 gi to taxon_id pairs DBD::Oracle::db prepare failed: ORA-00942: table or view does not exist (DBD ERROR: OCIStmtExecute/Describe) at /var/local/gus_home/lib/perl/GUS/ObjRelP/DbiDbHandle.pm line 78. prepareAndExecute FAILED: GUS::ObjRelP::DbiDbHandle=HASH(0x86c9d48)->errstr Thanks, Chetna __________________________________ Do you Yahoo!? Free Pop-Up Blocker - Get it now http://companion.yahoo.com/ |
|
From: Fidel S. <fi...@vb...> - 2003-12-03 16:16:21
|
Steve,
No, there is not a performance problem.
Fidel
Steve Fischer <st...@pc...> writes:
> Fidel-
> Thanks for the profiler info.
> About the multiple connections. I agree that ideally it should be one
> connection. But, there are some non-obvious issues around this, such
> as keeping independent transactions alive, that i don't fully
> understand. Two of the connections seem to be caused by ga and the
> rest by the object layer. When we renovate the object layer i will
> pay attention to this. But for now, besides the fact that it is
> suspicious and wasteful, is there any particular performance problem
> it is causing?
> steve
> Fidel Salas wrote:
>
> Steve,
>
> The profiler I used is Devel::DProf. It is run like this:
>
> $ perl -d:DProf ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDat
> abase --attrlist external_database_id,name,lowercase_name --valuelist "200^^^Ge
> nBank(main)^^^genbank(main)"
>
> The profile output is written to a file called 'tmon.out.' This file is interp
> reted
> by the program 'dprofpp.'
>
>>From your sample output below, it seems like it is the same profiler you used.
>
> There were two main points to my original post:
>
> 1. The dbiDsn issue. I do not know what your oracle setup is at CBIL or what
> it is at
> other places where people are using GUS. As I pointed out before, we connect t
> o oracle
> from a client machine, that is, the oracle server and client are on different m
> achines.
> Oracle has multiple ways of making connections (using environment variables or
> tnsname.ora)
> and I am not sure whether the server/client configuration impacts what the dsn
> must be in
> order for things to work properly.
>
> 2. Connecting to the database (by SubmitRow) multiple times. The reason I spo
> ke to you
> (Steve) about how slow things were running has to do with the multiple connecti
> ons to
> the database. I probably would have ignored the issue (due to my limited time
> at VBI)
> if SubmitRow had been taking 3 minutes to run, but since it was taking 12 min.
> or more,
> it became a concern. There is no reason for SubmitRow or any other plugin to h
> ave to
> connect to the database more than once per invocation. In the course of debugg
> ing the
> problem, I placed a few print statements in various places. Here is the output
> after
> one invocation:
>
> gus@tyler-lab:/home/apps/gus/debug$ ga GUS::Common::Plugin::SubmitRow
> --tablename SRes::ExternalDatabase --attrlist external_database_id,name,lowerc
> ase_name --valuelist "200^^^GenBank(main)^^^genbank(main)"
>
> Creating GusApplication Object
> Reading properties from /home/apps/gus/gushome/config/GUS-PluginMgr.pro
> p
>
> Created GusApplication object
>
> Entering GA parseAndRun
>
> Entering doMajorMode
>
> Entering doMajorMode_Run
>
> I am in GusApplication module. Entering connect_to_database method
>
> PlugIn is:GUS::PluginMgr::GusApplication=HASH(0x813361c)
> Reading properties from /home/apps/gus/.gus.properties
>
> My login is:GUSrw core is:Core dbiDsn is:dbi:Oracle:host=tuor.
> bioinformatics.vt.edu;sid=gusdev oraDfltRbs is:none
>
> Creating DbiDatabase
>
> Created DbiDatabase
>
> *************I am in DbiDbHandle and about to connect to database with
> following parameters:
> class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tu
> or.bioinformatics.vt.edu;sid=gusdev
> user: GUSrw autoCommit: *****************
>
> I am in GusApplication module. Entering connect_to_database method
>
> PlugIn is:GUS::Common::Plugin::SubmitRow=HASH(0x8143fb4)
>
> My login is:GUSrw core is:Core dbiDsn is:dbi:Oracle:host=tuor.
> bioinformatics.vt.edu;sid=gusdev oraDfltRbs is:none
>
> Creating DbiDatabase
>
> Created DbiDatabase
>
> *************I am in DbiDbHandle and about to connect to database with
> following parameters:
> class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tu
> or.bioinformatics.vt.edu;sid=gusdev
> user: GUSrw autoCommit: *****************
>
> *************I am in DbiDbHandle and about to connect to database with
> following parameters:
> class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tu
> or.bioinformatics.vt.edu;sid=gusdev
> user: GUSrw autoCommit: 1 *****************
>
> Creating DbiTable object
>
> Created DbiTable object
>
> *************I am in DbiDbHandle and about to connect to database with
> following parameters:
> class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tu
> or.bioinformatics.vt.edu;sid=gusdev
> user: GUSrw autoCommit: *****************
>
> Creating DbiTable object
>
> Created DbiTable object
>
> Entering openInvocation
>
> PlugIn is: GUS::Common::Plugin::SubmitRow=HASH(0x8143fb4)
>
> Creating DbiTable object
>
> Created DbiTable object
>
> Creating DbiTable object
>
> Created DbiTable object
>
> Creating DbiTable object
>
> Created DbiTable object
>
> Creating DbiTable object
>
> Created DbiTable object
> Mon Nov 10 10:44:14 2003 ALGINVID 10
> Mon Nov 10 10:44:14 2003 COMMIT commit off
>
> Creating DbiTable object
>
> Created DbiTable object
> Mon Nov 10 10:44:14 2003 Row updated
> Mon Nov 10 10:44:14 2003 RESULT Updated one row
> gus@tyler-lab:/home/apps/gus/debug$
>
> Connecting to the database is expensive and it is, obviously, desirable to mini
> mize
> the number of connections made.
>
> Fidel
>
>
> Steve Fischer <st...@pc...> writes:
>
> > Folks-
>
> > I ran the SubmitRow plugin command mentioned below, with this dsn:
>
> > dbiDsn=dbi:Oracle:host=cbilbld.pcbi.upenn.edu;sid=cbilbld
>
> > It took less than a second. So, whatever the problem is on Fidel's
> > site is not reproducable here. I will forward the dsn question to our
> > SA folks. Maybe they'll have a clue.
>
> > As far as the surprising profile: much of it is explained by the call
> > to GUS::ObjRelP::DbiDatabase::cacheTableNames which builds a cache of
> > table names. This is a bit of object layer overhead. There are about
> > 900 tables in our GUS's TableInfo table. The real mystery is why this
> > is in one profile but not in the other:
>
> > 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTa
> ble
>
>
>
>
> > Fidel- what tool did you use for the profiling?
>
> > Steve
>
>
>
>
>
> > Fidel Salas wrote:
>
>
> Hi folks!
>
> This message is mostly with the hope that it may be of benefit
> to those experiencing performance issues running plugins. I also
> ask two questions at the end related to implementation of the
> plugins, specifically the SubmitRow plugin.
>
> We had been experiencing performance issues when running plugins
> and discovered that it was because of the dbiDsn string we were
> using in the .gus.properties file.
>
> First, let me describe our environment:
> Our oracle is 9i with the server running on a big Sun machine
> and the client running on linux, with following:
> DBD::Oracle : 1.14
> DBI : 1.38
> OS : linux
> Perl : 5.008
>
> The dbiDsn causing the slow performance follows the same form as
> the one found in the gus.properties.sample in the GUS distribution
> (in install directory). In our case, that dbiDsn is:
> dbiDsn=dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev
>
> Following is the result of profiling the following call:
> ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase\
> --attrlist external_database_id,name,lowercase_name\
> --valuelist "200^^^GenBank(main)^^^genbank(main)"
> Total Elapsed Time = 757.1146 Seconds
> User+System Time = 0.484621 Seconds
> Exclusive Times
> %Time ExclSec CumulS #Calls sec/call Csec/c Name
> 12.3 0.060 0.208 11 0.0054 0.0189 GUS::PluginMgr::GusApplication::BEG
> IN
> 8.25 0.040 0.040 4 0.0100 0.0100 DBD::Oracle::db::_login
> 6.19 0.030 0.050 3 0.0100 0.0166 GUS::Model::Core::Algorithm_Row::BE
> GIN
> 6.19 0.030 0.021 914 0.0000 0.0000 GUS::ObjRelP::DbiTable::getFullClas
> sName
> 6.19 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN
> 4.13 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file
> 4.13 0.020 0.020 37 0.0005 0.0005 DBI::_setup_handle
> 4.13 0.020 0.020 28 0.0007 0.0007 DBI::st::execute
> 4.13 0.020 0.020 6 0.0033 0.0033 GUS::Model::GusRow::BEGIN
> 4.13 0.020 0.078 37 0.0005 0.0021 GUS::ObjRelP::DbiDatabase::getTable
> 4.13 0.020 0.042 11 0.0018 0.0038 GUS::ObjRelP::DbiTable::BEGIN
> 4.13 0.020 0.238 3 0.0067 0.0793 main::BEGIN
> 4.13 0.020 0.069 4 0.0050 0.0173 GUS::ObjRelP::DbiDatabase::BEGIN
> 3.71 0.018 0.026 1 0.0183 0.0264 GUS::ObjRelP::DbiDatabase::cacheTab
> leNames
> 2.06 0.010 0.008 870 0.0000 0.0000 DBI::st::fetchrow_array
>
> When I change the dbiDsn string to:
> dbiDsn=dbi:Oracle:gusdev.bioinformatics.vt.edu
> profiling results in:
>
> Total Elapsed Time = 1.104655 Seconds
> User+System Time = 0.534655 Seconds
> Exclusive Times
> %Time ExclSec CumulS #Calls sec/call Csec/c Name
> 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable
> 7.48 0.040 0.050 3 0.0133 0.0166 GUS::Model::Core::Algorithm_Row::BE
> GIN
> 7.48 0.040 0.208 11 0.0036 0.0189 GUS::PluginMgr::GusApplication::BEG
> IN
> 5.61 0.030 0.030 4 0.0075 0.0075 DBD::Oracle::db::_login
> 5.61 0.030 0.028 870 0.0000 0.0000 DBI::st::fetchrow_array
> 5.61 0.030 0.084 951 0.0000 0.0001 GUS::ObjRelP::DbiDatabase::getFullT
> ableClassName
> 5.61 0.030 0.109 15 0.0020 0.0073 GUS::PluginMgr::Plugin::BEGIN
> 5.61 0.030 0.118 37 0.0008 0.0032 GUS::ObjRelP::DbiDatabase::getTable
> 5.61 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN
> 3.74 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file
> 3.74 0.020 0.238 3 0.0067 0.0793 main::BEGIN
> 3.37 0.018 0.056 1 0.0183 0.0564 GUS::ObjRelP::DbiDatabase::cacheTab
> leNames
> 1.87 0.010 0.010 30 0.0003 0.0003 Exporter::import
> 1.87 0.010 0.010 15 0.0007 0.0007 CBIL::Util::PropertySet::getProp
> 1.87 0.010 0.010 37 0.0003 0.0003 DBI::st::TIEHASH
>
> As you can see, that's a difference of > 12 minutes vs a little more than 1 sec
> ond.
>
> Question 1:
> Now, looking at the profile results and walking over the code, I see
> that there are several connects made to the database (4 logins shown
> above) with the same parameters.
> An obvious question, is why are several connections made?
>
> Question 2:
> Looking at the profiles above, does it make sense for such a simple submit row
> plugin
> to make so many calls to GUS::ObjRelP::DbiRow::getTable, DBI::st::fetchrow_arra
> y,
> GUS::ObjRelP::DbiDatabase::getFullTableClassName, and others?
>
> Regards
>
> Fidel
>
>
>
> -------------------------------------------------------
> This SF.Net email sponsored by: ApacheCon 2003,
> 16-19 November in Las Vegas. Learn firsthand the latest
> developments in Apache, PHP, Perl, XML, Java, MySQL,
> WebDAV, and more! http://www.apachecon.com/
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>
>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> Does SourceForge.net help you be more productive? Does it
> help you create better code? SHARE THE LOVE, and help us help
> YOU! Click Here: http://sourceforge.net/donate/
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
|
|
From: Steve F. <st...@pc...> - 2003-12-03 16:08:05
|
Fidel- Thanks for the profiler info. About the multiple connections. I agree that ideally it should be one connection. But, there are some non-obvious issues around this, such as keeping independent transactions alive, that i don't fully understand. Two of the connections seem to be caused by ga and the rest by the object layer. When we renovate the object layer i will pay attention to this. But for now, besides the fact that it is suspicious and wasteful, is there any particular performance problem it is causing? steve Fidel Salas wrote: >Steve, > >The profiler I used is Devel::DProf. It is run like this: > >$ perl -d:DProf ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase --attrlist external_database_id,name,lowercase_name --valuelist "200^^^GenBank(main)^^^genbank(main)" > >The profile output is written to a file called 'tmon.out.' This file is interpreted >by the program 'dprofpp.' > >>From your sample output below, it seems like it is the same profiler you used. > >There were two main points to my original post: > >1. The dbiDsn issue. I do not know what your oracle setup is at CBIL or what it is at >other places where people are using GUS. As I pointed out before, we connect to oracle >from a client machine, that is, the oracle server and client are on different machines. >Oracle has multiple ways of making connections (using environment variables or tnsname.ora) >and I am not sure whether the server/client configuration impacts what the dsn must be in >order for things to work properly. > >2. Connecting to the database (by SubmitRow) multiple times. The reason I spoke to you >(Steve) about how slow things were running has to do with the multiple connections to >the database. I probably would have ignored the issue (due to my limited time at VBI) >if SubmitRow had been taking 3 minutes to run, but since it was taking 12 min. or more, >it became a concern. There is no reason for SubmitRow or any other plugin to have to >connect to the database more than once per invocation. In the course of debugging the >problem, I placed a few print statements in various places. Here is the output after >one invocation: > > gus@tyler-lab:/home/apps/gus/debug$ ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase --attrlist external_database_id,name,lowercase_name --valuelist "200^^^GenBank(main)^^^genbank(main)" > > Creating GusApplication Object > Reading properties from /home/apps/gus/gushome/config/GUS-PluginMgr.prop > > Created GusApplication object > > Entering GA parseAndRun > > Entering doMajorMode > > Entering doMajorMode_Run > > I am in GusApplication module. Entering connect_to_database method > > PlugIn is:GUS::PluginMgr::GusApplication=HASH(0x813361c) > Reading properties from /home/apps/gus/.gus.properties > > My login is:GUSrw core is:Core dbiDsn is:dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev oraDfltRbs is:none > > Creating DbiDatabase > > Created DbiDatabase > > *************I am in DbiDbHandle and about to connect to database with following parameters: > class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev > user: GUSrw autoCommit: ***************** > > I am in GusApplication module. Entering connect_to_database method > > PlugIn is:GUS::Common::Plugin::SubmitRow=HASH(0x8143fb4) > > My login is:GUSrw core is:Core dbiDsn is:dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev oraDfltRbs is:none > > Creating DbiDatabase > > Created DbiDatabase > > *************I am in DbiDbHandle and about to connect to database with following parameters: > class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev > user: GUSrw autoCommit: ***************** > > *************I am in DbiDbHandle and about to connect to database with following parameters: > class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev > user: GUSrw autoCommit: 1 ***************** > > Creating DbiTable object > > Created DbiTable object > > *************I am in DbiDbHandle and about to connect to database with following parameters: > class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev > user: GUSrw autoCommit: ***************** > > Creating DbiTable object > > Created DbiTable object > > Entering openInvocation > > PlugIn is: GUS::Common::Plugin::SubmitRow=HASH(0x8143fb4) > > Creating DbiTable object > > Created DbiTable object > > Creating DbiTable object > > Created DbiTable object > > Creating DbiTable object > > Created DbiTable object > > Creating DbiTable object > > Created DbiTable object > Mon Nov 10 10:44:14 2003 ALGINVID 10 > Mon Nov 10 10:44:14 2003 COMMIT commit off > > Creating DbiTable object > > Created DbiTable object > Mon Nov 10 10:44:14 2003 Row updated > Mon Nov 10 10:44:14 2003 RESULT Updated one row > gus@tyler-lab:/home/apps/gus/debug$ > >Connecting to the database is expensive and it is, obviously, desirable to minimize >the number of connections made. > >Fidel > > >Steve Fischer <st...@pc...> writes: > > > Folks- > > > > I ran the SubmitRow plugin command mentioned below, with this dsn: > > > > dbiDsn=dbi:Oracle:host=cbilbld.pcbi.upenn.edu;sid=cbilbld > > > > It took less than a second. So, whatever the problem is on Fidel's > > site is not reproducable here. I will forward the dsn question to our > > SA folks. Maybe they'll have a clue. > > > > As far as the surprising profile: much of it is explained by the call > > to GUS::ObjRelP::DbiDatabase::cacheTableNames which builds a cache of > > table names. This is a bit of object layer overhead. There are about > > 900 tables in our GUS's TableInfo table. The real mystery is why this > > is in one profile but not in the other: > > > > 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable > > >> >> > > Fidel- what tool did you use for the profiling? > > > > Steve > > >> >> >> > > Fidel Salas wrote: > > >>>Hi folks! >>> >>>This message is mostly with the hope that it may be of benefit >>>to those experiencing performance issues running plugins. I also >>>ask two questions at the end related to implementation of the >>>plugins, specifically the SubmitRow plugin. >>> >>>We had been experiencing performance issues when running plugins >>>and discovered that it was because of the dbiDsn string we were >>>using in the .gus.properties file. >>> >>>First, let me describe our environment: >>>Our oracle is 9i with the server running on a big Sun machine >>>and the client running on linux, with following: >>>DBD::Oracle : 1.14 >>> DBI : 1.38 >>> OS : linux >>> Perl : 5.008 >>> >>>The dbiDsn causing the slow performance follows the same form as >>>the one found in the gus.properties.sample in the GUS distribution >>>(in install directory). In our case, that dbiDsn is: >>>dbiDsn=dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev >>> >>>Following is the result of profiling the following call: >>>ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase\ >>>--attrlist external_database_id,name,lowercase_name\ >>>--valuelist "200^^^GenBank(main)^^^genbank(main)" >>>Total Elapsed Time = 757.1146 Seconds >>> User+System Time = 0.484621 Seconds >>>Exclusive Times >>>%Time ExclSec CumulS #Calls sec/call Csec/c Name >>>12.3 0.060 0.208 11 0.0054 0.0189 GUS::PluginMgr::GusApplication::BEGIN >>>8.25 0.040 0.040 4 0.0100 0.0100 DBD::Oracle::db::_login >>>6.19 0.030 0.050 3 0.0100 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN >>>6.19 0.030 0.021 914 0.0000 0.0000 GUS::ObjRelP::DbiTable::getFullClassName >>>6.19 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN >>>4.13 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file >>>4.13 0.020 0.020 37 0.0005 0.0005 DBI::_setup_handle >>>4.13 0.020 0.020 28 0.0007 0.0007 DBI::st::execute >>>4.13 0.020 0.020 6 0.0033 0.0033 GUS::Model::GusRow::BEGIN >>>4.13 0.020 0.078 37 0.0005 0.0021 GUS::ObjRelP::DbiDatabase::getTable >>>4.13 0.020 0.042 11 0.0018 0.0038 GUS::ObjRelP::DbiTable::BEGIN >>>4.13 0.020 0.238 3 0.0067 0.0793 main::BEGIN >>>4.13 0.020 0.069 4 0.0050 0.0173 GUS::ObjRelP::DbiDatabase::BEGIN >>>3.71 0.018 0.026 1 0.0183 0.0264 GUS::ObjRelP::DbiDatabase::cacheTableNames >>>2.06 0.010 0.008 870 0.0000 0.0000 DBI::st::fetchrow_array >>> >>>When I change the dbiDsn string to: >>>dbiDsn=dbi:Oracle:gusdev.bioinformatics.vt.edu >>>profiling results in: >>> >>>Total Elapsed Time = 1.104655 Seconds >>> User+System Time = 0.534655 Seconds >>>Exclusive Times >>>%Time ExclSec CumulS #Calls sec/call Csec/c Name >>>11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable >>>7.48 0.040 0.050 3 0.0133 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN >>>7.48 0.040 0.208 11 0.0036 0.0189 GUS::PluginMgr::GusApplication::BEGIN >>>5.61 0.030 0.030 4 0.0075 0.0075 DBD::Oracle::db::_login >>>5.61 0.030 0.028 870 0.0000 0.0000 DBI::st::fetchrow_array >>>5.61 0.030 0.084 951 0.0000 0.0001 GUS::ObjRelP::DbiDatabase::getFullTableClassName >>>5.61 0.030 0.109 15 0.0020 0.0073 GUS::PluginMgr::Plugin::BEGIN >>>5.61 0.030 0.118 37 0.0008 0.0032 GUS::ObjRelP::DbiDatabase::getTable >>>5.61 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN >>>3.74 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file >>>3.74 0.020 0.238 3 0.0067 0.0793 main::BEGIN >>>3.37 0.018 0.056 1 0.0183 0.0564 GUS::ObjRelP::DbiDatabase::cacheTableNames >>>1.87 0.010 0.010 30 0.0003 0.0003 Exporter::import >>>1.87 0.010 0.010 15 0.0007 0.0007 CBIL::Util::PropertySet::getProp >>>1.87 0.010 0.010 37 0.0003 0.0003 DBI::st::TIEHASH >>> >>>As you can see, that's a difference of > 12 minutes vs a little more than 1 second. >>> >>>Question 1: >>>Now, looking at the profile results and walking over the code, I see >>>that there are several connects made to the database (4 logins shown >>>above) with the same parameters. >>>An obvious question, is why are several connections made? >>> >>>Question 2: >>>Looking at the profiles above, does it make sense for such a simple submit row plugin >>>to make so many calls to GUS::ObjRelP::DbiRow::getTable, DBI::st::fetchrow_array, >>>GUS::ObjRelP::DbiDatabase::getFullTableClassName, and others? >>> >>>Regards >>> >>>Fidel >>> >>> >>> >>>------------------------------------------------------- >>>This SF.Net email sponsored by: ApacheCon 2003, >>>16-19 November in Las Vegas. Learn firsthand the latest >>>developments in Apache, PHP, Perl, XML, Java, MySQL, >>>WebDAV, and more! http://www.apachecon.com/ >>>_______________________________________________ >>>Gusdev-gusdev mailing list >>>Gus...@li... >>>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev >>> >>> >>> > > > >------------------------------------------------------- >This SF.net email is sponsored by: SF.net Giveback Program. >Does SourceForge.net help you be more productive? Does it >help you create better code? SHARE THE LOVE, and help us help >YOU! Click Here: http://sourceforge.net/donate/ >_______________________________________________ >Gusdev-gusdev mailing list >Gus...@li... >https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev > > |
|
From: Fidel S. <fi...@vb...> - 2003-12-03 15:52:07
|
Steve, The profiler I used is Devel::DProf. It is run like this: $ perl -d:DProf ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase --attrlist external_database_id,name,lowercase_name --valuelist "200^^^GenBank(main)^^^genbank(main)" The profile output is written to a file called 'tmon.out.' This file is interpreted by the program 'dprofpp.' From your sample output below, it seems like it is the same profiler you used. There were two main points to my original post: 1. The dbiDsn issue. I do not know what your oracle setup is at CBIL or what it is at other places where people are using GUS. As I pointed out before, we connect to oracle from a client machine, that is, the oracle server and client are on different machines. Oracle has multiple ways of making connections (using environment variables or tnsname.ora) and I am not sure whether the server/client configuration impacts what the dsn must be in order for things to work properly. 2. Connecting to the database (by SubmitRow) multiple times. The reason I spoke to you (Steve) about how slow things were running has to do with the multiple connections to the database. I probably would have ignored the issue (due to my limited time at VBI) if SubmitRow had been taking 3 minutes to run, but since it was taking 12 min. or more, it became a concern. There is no reason for SubmitRow or any other plugin to have to connect to the database more than once per invocation. In the course of debugging the problem, I placed a few print statements in various places. Here is the output after one invocation: gus@tyler-lab:/home/apps/gus/debug$ ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase --attrlist external_database_id,name,lowercase_name --valuelist "200^^^GenBank(main)^^^genbank(main)" Creating GusApplication Object Reading properties from /home/apps/gus/gushome/config/GUS-PluginMgr.prop Created GusApplication object Entering GA parseAndRun Entering doMajorMode Entering doMajorMode_Run I am in GusApplication module. Entering connect_to_database method PlugIn is:GUS::PluginMgr::GusApplication=HASH(0x813361c) Reading properties from /home/apps/gus/.gus.properties My login is:GUSrw core is:Core dbiDsn is:dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev oraDfltRbs is:none Creating DbiDatabase Created DbiDatabase *************I am in DbiDbHandle and about to connect to database with following parameters: class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev user: GUSrw autoCommit: ***************** I am in GusApplication module. Entering connect_to_database method PlugIn is:GUS::Common::Plugin::SubmitRow=HASH(0x8143fb4) My login is:GUSrw core is:Core dbiDsn is:dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev oraDfltRbs is:none Creating DbiDatabase Created DbiDatabase *************I am in DbiDbHandle and about to connect to database with following parameters: class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev user: GUSrw autoCommit: ***************** *************I am in DbiDbHandle and about to connect to database with following parameters: class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev user: GUSrw autoCommit: 1 ***************** Creating DbiTable object Created DbiTable object *************I am in DbiDbHandle and about to connect to database with following parameters: class: GUS::ObjRelP::DbiDbHandle dsn: dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev user: GUSrw autoCommit: ***************** Creating DbiTable object Created DbiTable object Entering openInvocation PlugIn is: GUS::Common::Plugin::SubmitRow=HASH(0x8143fb4) Creating DbiTable object Created DbiTable object Creating DbiTable object Created DbiTable object Creating DbiTable object Created DbiTable object Creating DbiTable object Created DbiTable object Mon Nov 10 10:44:14 2003 ALGINVID 10 Mon Nov 10 10:44:14 2003 COMMIT commit off Creating DbiTable object Created DbiTable object Mon Nov 10 10:44:14 2003 Row updated Mon Nov 10 10:44:14 2003 RESULT Updated one row gus@tyler-lab:/home/apps/gus/debug$ Connecting to the database is expensive and it is, obviously, desirable to minimize the number of connections made. Fidel Steve Fischer <st...@pc...> writes: > Folks- > > I ran the SubmitRow plugin command mentioned below, with this dsn: > > dbiDsn=dbi:Oracle:host=cbilbld.pcbi.upenn.edu;sid=cbilbld > > It took less than a second. So, whatever the problem is on Fidel's > site is not reproducable here. I will forward the dsn question to our > SA folks. Maybe they'll have a clue. > > As far as the surprising profile: much of it is explained by the call > to GUS::ObjRelP::DbiDatabase::cacheTableNames which builds a cache of > table names. This is a bit of object layer overhead. There are about > 900 tables in our GUS's TableInfo table. The real mystery is why this > is in one profile but not in the other: > > 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable > > > Fidel- what tool did you use for the profiling? > > Steve > > > > Fidel Salas wrote: > >>Hi folks! >> >>This message is mostly with the hope that it may be of benefit >>to those experiencing performance issues running plugins. I also >>ask two questions at the end related to implementation of the >>plugins, specifically the SubmitRow plugin. >> >>We had been experiencing performance issues when running plugins >>and discovered that it was because of the dbiDsn string we were >>using in the .gus.properties file. >> >>First, let me describe our environment: >>Our oracle is 9i with the server running on a big Sun machine >>and the client running on linux, with following: >>DBD::Oracle : 1.14 >> DBI : 1.38 >> OS : linux >> Perl : 5.008 >> >>The dbiDsn causing the slow performance follows the same form as >>the one found in the gus.properties.sample in the GUS distribution >>(in install directory). In our case, that dbiDsn is: >>dbiDsn=dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev >> >>Following is the result of profiling the following call: >>ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase\ >> --attrlist external_database_id,name,lowercase_name\ >> --valuelist "200^^^GenBank(main)^^^genbank(main)" >> Total Elapsed Time = 757.1146 Seconds >> User+System Time = 0.484621 Seconds >>Exclusive Times >>%Time ExclSec CumulS #Calls sec/call Csec/c Name >> 12.3 0.060 0.208 11 0.0054 0.0189 GUS::PluginMgr::GusApplication::BEGIN >> 8.25 0.040 0.040 4 0.0100 0.0100 DBD::Oracle::db::_login >> 6.19 0.030 0.050 3 0.0100 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN >> 6.19 0.030 0.021 914 0.0000 0.0000 GUS::ObjRelP::DbiTable::getFullClassName >> 6.19 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN >> 4.13 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file >> 4.13 0.020 0.020 37 0.0005 0.0005 DBI::_setup_handle >> 4.13 0.020 0.020 28 0.0007 0.0007 DBI::st::execute >> 4.13 0.020 0.020 6 0.0033 0.0033 GUS::Model::GusRow::BEGIN >> 4.13 0.020 0.078 37 0.0005 0.0021 GUS::ObjRelP::DbiDatabase::getTable >> 4.13 0.020 0.042 11 0.0018 0.0038 GUS::ObjRelP::DbiTable::BEGIN >> 4.13 0.020 0.238 3 0.0067 0.0793 main::BEGIN >> 4.13 0.020 0.069 4 0.0050 0.0173 GUS::ObjRelP::DbiDatabase::BEGIN >> 3.71 0.018 0.026 1 0.0183 0.0264 GUS::ObjRelP::DbiDatabase::cacheTableNames >> 2.06 0.010 0.008 870 0.0000 0.0000 DBI::st::fetchrow_array >> >>When I change the dbiDsn string to: >>dbiDsn=dbi:Oracle:gusdev.bioinformatics.vt.edu >>profiling results in: >> >>Total Elapsed Time = 1.104655 Seconds >> User+System Time = 0.534655 Seconds >>Exclusive Times >>%Time ExclSec CumulS #Calls sec/call Csec/c Name >> 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable >> 7.48 0.040 0.050 3 0.0133 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN >> 7.48 0.040 0.208 11 0.0036 0.0189 GUS::PluginMgr::GusApplication::BEGIN >> 5.61 0.030 0.030 4 0.0075 0.0075 DBD::Oracle::db::_login >> 5.61 0.030 0.028 870 0.0000 0.0000 DBI::st::fetchrow_array >> 5.61 0.030 0.084 951 0.0000 0.0001 GUS::ObjRelP::DbiDatabase::getFullTableClassName >> 5.61 0.030 0.109 15 0.0020 0.0073 GUS::PluginMgr::Plugin::BEGIN >> 5.61 0.030 0.118 37 0.0008 0.0032 GUS::ObjRelP::DbiDatabase::getTable >> 5.61 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN >> 3.74 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file >> 3.74 0.020 0.238 3 0.0067 0.0793 main::BEGIN >> 3.37 0.018 0.056 1 0.0183 0.0564 GUS::ObjRelP::DbiDatabase::cacheTableNames >> 1.87 0.010 0.010 30 0.0003 0.0003 Exporter::import >> 1.87 0.010 0.010 15 0.0007 0.0007 CBIL::Util::PropertySet::getProp >> 1.87 0.010 0.010 37 0.0003 0.0003 DBI::st::TIEHASH >> >>As you can see, that's a difference of > 12 minutes vs a little more than 1 second. >> >>Question 1: >> Now, looking at the profile results and walking over the code, I see >> that there are several connects made to the database (4 logins shown >> above) with the same parameters. >>An obvious question, is why are several connections made? >> >>Question 2: >>Looking at the profiles above, does it make sense for such a simple submit row plugin >>to make so many calls to GUS::ObjRelP::DbiRow::getTable, DBI::st::fetchrow_array, >>GUS::ObjRelP::DbiDatabase::getFullTableClassName, and others? >> >>Regards >> >>Fidel >> >> >> >>------------------------------------------------------- >>This SF.Net email sponsored by: ApacheCon 2003, >>16-19 November in Las Vegas. Learn firsthand the latest >>developments in Apache, PHP, Perl, XML, Java, MySQL, >>WebDAV, and more! http://www.apachecon.com/ >>_______________________________________________ >>Gusdev-gusdev mailing list >>Gus...@li... >>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev >> |
|
From: Steve F. <st...@pc...> - 2003-12-03 13:42:39
|
Folks- I ran the SubmitRow plugin command mentioned below, with this dsn: dbiDsn=dbi:Oracle:host=cbilbld.pcbi.upenn.edu;sid=cbilbld It took less than a second. So, whatever the problem is on Fidel's site is not reproducable here. I will forward the dsn question to our SA folks. Maybe they'll have a clue. As far as the surprising profile: much of it is explained by the call to GUS::ObjRelP::DbiDatabase::cacheTableNames which builds a cache of table names. This is a bit of object layer overhead. There are about 900 tables in our GUS's TableInfo table. The real mystery is why this is in one profile but not in the other: 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable Fidel- what tool did you use for the profiling? Steve Fidel Salas wrote: >Hi folks! > >This message is mostly with the hope that it may be of benefit >to those experiencing performance issues running plugins. I also >ask two questions at the end related to implementation of the >plugins, specifically the SubmitRow plugin. > >We had been experiencing performance issues when running plugins >and discovered that it was because of the dbiDsn string we were >using in the .gus.properties file. > >First, let me describe our environment: >Our oracle is 9i with the server running on a big Sun machine >and the client running on linux, with following: >DBD::Oracle : 1.14 > DBI : 1.38 > OS : linux > Perl : 5.008 > >The dbiDsn causing the slow performance follows the same form as >the one found in the gus.properties.sample in the GUS distribution >(in install directory). In our case, that dbiDsn is: >dbiDsn=dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev > >Following is the result of profiling the following call: >ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase\ > --attrlist external_database_id,name,lowercase_name\ > --valuelist "200^^^GenBank(main)^^^genbank(main)" > >Total Elapsed Time = 757.1146 Seconds > User+System Time = 0.484621 Seconds >Exclusive Times >%Time ExclSec CumulS #Calls sec/call Csec/c Name > 12.3 0.060 0.208 11 0.0054 0.0189 GUS::PluginMgr::GusApplication::BEGIN > 8.25 0.040 0.040 4 0.0100 0.0100 DBD::Oracle::db::_login > 6.19 0.030 0.050 3 0.0100 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN > 6.19 0.030 0.021 914 0.0000 0.0000 GUS::ObjRelP::DbiTable::getFullClassName > 6.19 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN > 4.13 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file > 4.13 0.020 0.020 37 0.0005 0.0005 DBI::_setup_handle > 4.13 0.020 0.020 28 0.0007 0.0007 DBI::st::execute > 4.13 0.020 0.020 6 0.0033 0.0033 GUS::Model::GusRow::BEGIN > 4.13 0.020 0.078 37 0.0005 0.0021 GUS::ObjRelP::DbiDatabase::getTable > 4.13 0.020 0.042 11 0.0018 0.0038 GUS::ObjRelP::DbiTable::BEGIN > 4.13 0.020 0.238 3 0.0067 0.0793 main::BEGIN > 4.13 0.020 0.069 4 0.0050 0.0173 GUS::ObjRelP::DbiDatabase::BEGIN > 3.71 0.018 0.026 1 0.0183 0.0264 GUS::ObjRelP::DbiDatabase::cacheTableNames > 2.06 0.010 0.008 870 0.0000 0.0000 DBI::st::fetchrow_array > >When I change the dbiDsn string to: >dbiDsn=dbi:Oracle:gusdev.bioinformatics.vt.edu >profiling results in: > >Total Elapsed Time = 1.104655 Seconds > User+System Time = 0.534655 Seconds >Exclusive Times >%Time ExclSec CumulS #Calls sec/call Csec/c Name > 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable > 7.48 0.040 0.050 3 0.0133 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN > 7.48 0.040 0.208 11 0.0036 0.0189 GUS::PluginMgr::GusApplication::BEGIN > 5.61 0.030 0.030 4 0.0075 0.0075 DBD::Oracle::db::_login > 5.61 0.030 0.028 870 0.0000 0.0000 DBI::st::fetchrow_array > 5.61 0.030 0.084 951 0.0000 0.0001 GUS::ObjRelP::DbiDatabase::getFullTableClassName > 5.61 0.030 0.109 15 0.0020 0.0073 GUS::PluginMgr::Plugin::BEGIN > 5.61 0.030 0.118 37 0.0008 0.0032 GUS::ObjRelP::DbiDatabase::getTable > 5.61 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN > 3.74 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file > 3.74 0.020 0.238 3 0.0067 0.0793 main::BEGIN > 3.37 0.018 0.056 1 0.0183 0.0564 GUS::ObjRelP::DbiDatabase::cacheTableNames > 1.87 0.010 0.010 30 0.0003 0.0003 Exporter::import > 1.87 0.010 0.010 15 0.0007 0.0007 CBIL::Util::PropertySet::getProp > 1.87 0.010 0.010 37 0.0003 0.0003 DBI::st::TIEHASH > >As you can see, that's a difference of > 12 minutes vs a little more than 1 second. > >Question 1: >Now, looking at the profile results and walking over the code, I see that there are >several connects made to the database (4 logins shown above) with the same parameters. >An obvious question, is why are several connections made? > >Question 2: >Looking at the profiles above, does it make sense for such a simple submit row plugin >to make so many calls to GUS::ObjRelP::DbiRow::getTable, DBI::st::fetchrow_array, >GUS::ObjRelP::DbiDatabase::getFullTableClassName, and others? > >Regards > >Fidel > > > >------------------------------------------------------- >This SF.Net email sponsored by: ApacheCon 2003, >16-19 November in Las Vegas. Learn firsthand the latest >developments in Apache, PHP, Perl, XML, Java, MySQL, >WebDAV, and more! http://www.apachecon.com/ >_______________________________________________ >Gusdev-gusdev mailing list >Gus...@li... >https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev > > |
|
From: Steve F. <st...@pc...> - 2003-12-03 13:07:53
|
Folks- I somehow missed the original mail. Fidel- thanks for doing this analysis. Terry- thanks for reviving the issue. Will look into. Steve Terry Clark wrote: >Greetings, >I was watching with interest (and mulling over) >this performance report (posted 11-10-2003), >but didn't see anything come through gusdev. >Were there further findings? > >Terry > > > >On 0, Fidel Salas <fi...@vb...> wrote: > > >>Hi folks! >> >>This message is mostly with the hope that it may be of benefit >>to those experiencing performance issues running plugins. I also >>ask two questions at the end related to implementation of the >>plugins, specifically the SubmitRow plugin. >> >>We had been experiencing performance issues when running plugins >>and discovered that it was because of the dbiDsn string we were >>using in the .gus.properties file. >> >>First, let me describe our environment: >>Our oracle is 9i with the server running on a big Sun machine >>and the client running on linux, with following: >>DBD::Oracle : 1.14 >> DBI : 1.38 >> OS : linux >> Perl : 5.008 >> >>The dbiDsn causing the slow performance follows the same form as >>the one found in the gus.properties.sample in the GUS distribution >>(in install directory). In our case, that dbiDsn is: >>dbiDsn=dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev >> >>Following is the result of profiling the following call: >>ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase\ >> --attrlist external_database_id,name,lowercase_name\ >> --valuelist "200^^^GenBank(main)^^^genbank(main)" >> >>Total Elapsed Time = 757.1146 Seconds >> User+System Time = 0.484621 Seconds >>Exclusive Times >>%Time ExclSec CumulS #Calls sec/call Csec/c Name >> 12.3 0.060 0.208 11 0.0054 0.0189 GUS::PluginMgr::GusApplication::BEGIN >> 8.25 0.040 0.040 4 0.0100 0.0100 DBD::Oracle::db::_login >> 6.19 0.030 0.050 3 0.0100 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN >> 6.19 0.030 0.021 914 0.0000 0.0000 GUS::ObjRelP::DbiTable::getFullClassName >> 6.19 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN >> 4.13 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file >> 4.13 0.020 0.020 37 0.0005 0.0005 DBI::_setup_handle >> 4.13 0.020 0.020 28 0.0007 0.0007 DBI::st::execute >> 4.13 0.020 0.020 6 0.0033 0.0033 GUS::Model::GusRow::BEGIN >> 4.13 0.020 0.078 37 0.0005 0.0021 GUS::ObjRelP::DbiDatabase::getTable >> 4.13 0.020 0.042 11 0.0018 0.0038 GUS::ObjRelP::DbiTable::BEGIN >> 4.13 0.020 0.238 3 0.0067 0.0793 main::BEGIN >> 4.13 0.020 0.069 4 0.0050 0.0173 GUS::ObjRelP::DbiDatabase::BEGIN >> 3.71 0.018 0.026 1 0.0183 0.0264 GUS::ObjRelP::DbiDatabase::cacheTableNames >> 2.06 0.010 0.008 870 0.0000 0.0000 DBI::st::fetchrow_array >> >>When I change the dbiDsn string to: >>dbiDsn=dbi:Oracle:gusdev.bioinformatics.vt.edu >>profiling results in: >> >>Total Elapsed Time = 1.104655 Seconds >> User+System Time = 0.534655 Seconds >>Exclusive Times >>%Time ExclSec CumulS #Calls sec/call Csec/c Name >> 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable >> 7.48 0.040 0.050 3 0.0133 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN >> 7.48 0.040 0.208 11 0.0036 0.0189 GUS::PluginMgr::GusApplication::BEGIN >> 5.61 0.030 0.030 4 0.0075 0.0075 DBD::Oracle::db::_login >> 5.61 0.030 0.028 870 0.0000 0.0000 DBI::st::fetchrow_array >> 5.61 0.030 0.084 951 0.0000 0.0001 GUS::ObjRelP::DbiDatabase::getFullTableClassName >> 5.61 0.030 0.109 15 0.0020 0.0073 GUS::PluginMgr::Plugin::BEGIN >> 5.61 0.030 0.118 37 0.0008 0.0032 GUS::ObjRelP::DbiDatabase::getTable >> 5.61 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN >> 3.74 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file >> 3.74 0.020 0.238 3 0.0067 0.0793 main::BEGIN >> 3.37 0.018 0.056 1 0.0183 0.0564 GUS::ObjRelP::DbiDatabase::cacheTableNames >> 1.87 0.010 0.010 30 0.0003 0.0003 Exporter::import >> 1.87 0.010 0.010 15 0.0007 0.0007 CBIL::Util::PropertySet::getProp >> 1.87 0.010 0.010 37 0.0003 0.0003 DBI::st::TIEHASH >> >>As you can see, that's a difference of > 12 minutes vs a little more than 1 second. >> >>Question 1: >>Now, looking at the profile results and walking over the code, I see that there are >>several connects made to the database (4 logins shown above) with the same parameters. >>An obvious question, is why are several connections made? >> >>Question 2: >>Looking at the profiles above, does it make sense for such a simple submit row plugin >>to make so many calls to GUS::ObjRelP::DbiRow::getTable, DBI::st::fetchrow_array, >>GUS::ObjRelP::DbiDatabase::getFullTableClassName, and others? >> >>Regards >> >>Fidel >> >> >> >>------------------------------------------------------- >>This SF.Net email sponsored by: ApacheCon 2003, >>16-19 November in Las Vegas. Learn firsthand the latest >>developments in Apache, PHP, Perl, XML, Java, MySQL, >>WebDAV, and more! http://www.apachecon.com/ >>_______________________________________________ >>Gusdev-gusdev mailing list >>Gus...@li... >>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev >> >> > > >------------------------------------------------------- >This SF.net email is sponsored by: SF.net Giveback Program. >Does SourceForge.net help you be more productive? Does it >help you create better code? SHARE THE LOVE, and help us help >YOU! Click Here: http://sourceforge.net/donate/ >_______________________________________________ >Gusdev-gusdev mailing list >Gus...@li... >https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev > > |
|
From: Terry C. <tw...@cs...> - 2003-12-03 01:53:18
|
Greetings, I was watching with interest (and mulling over) this performance report (posted 11-10-2003), but didn't see anything come through gusdev. Were there further findings? Terry On 0, Fidel Salas <fi...@vb...> wrote: > Hi folks! > > This message is mostly with the hope that it may be of benefit > to those experiencing performance issues running plugins. I also > ask two questions at the end related to implementation of the > plugins, specifically the SubmitRow plugin. > > We had been experiencing performance issues when running plugins > and discovered that it was because of the dbiDsn string we were > using in the .gus.properties file. > > First, let me describe our environment: > Our oracle is 9i with the server running on a big Sun machine > and the client running on linux, with following: > DBD::Oracle : 1.14 > DBI : 1.38 > OS : linux > Perl : 5.008 > > The dbiDsn causing the slow performance follows the same form as > the one found in the gus.properties.sample in the GUS distribution > (in install directory). In our case, that dbiDsn is: > dbiDsn=dbi:Oracle:host=tuor.bioinformatics.vt.edu;sid=gusdev > > Following is the result of profiling the following call: > ga GUS::Common::Plugin::SubmitRow --tablename SRes::ExternalDatabase\ > --attrlist external_database_id,name,lowercase_name\ > --valuelist "200^^^GenBank(main)^^^genbank(main)" > > Total Elapsed Time = 757.1146 Seconds > User+System Time = 0.484621 Seconds > Exclusive Times > %Time ExclSec CumulS #Calls sec/call Csec/c Name > 12.3 0.060 0.208 11 0.0054 0.0189 GUS::PluginMgr::GusApplication::BEGIN > 8.25 0.040 0.040 4 0.0100 0.0100 DBD::Oracle::db::_login > 6.19 0.030 0.050 3 0.0100 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN > 6.19 0.030 0.021 914 0.0000 0.0000 GUS::ObjRelP::DbiTable::getFullClassName > 6.19 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN > 4.13 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file > 4.13 0.020 0.020 37 0.0005 0.0005 DBI::_setup_handle > 4.13 0.020 0.020 28 0.0007 0.0007 DBI::st::execute > 4.13 0.020 0.020 6 0.0033 0.0033 GUS::Model::GusRow::BEGIN > 4.13 0.020 0.078 37 0.0005 0.0021 GUS::ObjRelP::DbiDatabase::getTable > 4.13 0.020 0.042 11 0.0018 0.0038 GUS::ObjRelP::DbiTable::BEGIN > 4.13 0.020 0.238 3 0.0067 0.0793 main::BEGIN > 4.13 0.020 0.069 4 0.0050 0.0173 GUS::ObjRelP::DbiDatabase::BEGIN > 3.71 0.018 0.026 1 0.0183 0.0264 GUS::ObjRelP::DbiDatabase::cacheTableNames > 2.06 0.010 0.008 870 0.0000 0.0000 DBI::st::fetchrow_array > > When I change the dbiDsn string to: > dbiDsn=dbi:Oracle:gusdev.bioinformatics.vt.edu > profiling results in: > > Total Elapsed Time = 1.104655 Seconds > User+System Time = 0.534655 Seconds > Exclusive Times > %Time ExclSec CumulS #Calls sec/call Csec/c Name > 11.2 0.060 0.055 2470 0.0000 0.0000 GUS::ObjRelP::DbiRow::getTable > 7.48 0.040 0.050 3 0.0133 0.0166 GUS::Model::Core::Algorithm_Row::BEGIN > 7.48 0.040 0.208 11 0.0036 0.0189 GUS::PluginMgr::GusApplication::BEGIN > 5.61 0.030 0.030 4 0.0075 0.0075 DBD::Oracle::db::_login > 5.61 0.030 0.028 870 0.0000 0.0000 DBI::st::fetchrow_array > 5.61 0.030 0.084 951 0.0000 0.0001 GUS::ObjRelP::DbiDatabase::getFullTableClassName > 5.61 0.030 0.109 15 0.0020 0.0073 GUS::PluginMgr::Plugin::BEGIN > 5.61 0.030 0.118 37 0.0008 0.0032 GUS::ObjRelP::DbiDatabase::getTable > 5.61 0.030 0.039 5 0.0059 0.0078 GUS::ObjRelP::DbiDbHandle::BEGIN > 3.74 0.020 0.020 3 0.0067 0.0067 DynaLoader::dl_load_file > 3.74 0.020 0.238 3 0.0067 0.0793 main::BEGIN > 3.37 0.018 0.056 1 0.0183 0.0564 GUS::ObjRelP::DbiDatabase::cacheTableNames > 1.87 0.010 0.010 30 0.0003 0.0003 Exporter::import > 1.87 0.010 0.010 15 0.0007 0.0007 CBIL::Util::PropertySet::getProp > 1.87 0.010 0.010 37 0.0003 0.0003 DBI::st::TIEHASH > > As you can see, that's a difference of > 12 minutes vs a little more than 1 second. > > Question 1: > Now, looking at the profile results and walking over the code, I see that there are > several connects made to the database (4 logins shown above) with the same parameters. > An obvious question, is why are several connections made? > > Question 2: > Looking at the profiles above, does it make sense for such a simple submit row plugin > to make so many calls to GUS::ObjRelP::DbiRow::getTable, DBI::st::fetchrow_array, > GUS::ObjRelP::DbiDatabase::getFullTableClassName, and others? > > Regards > > Fidel > > > > ------------------------------------------------------- > This SF.Net email sponsored by: ApacheCon 2003, > 16-19 November in Las Vegas. Learn firsthand the latest > developments in Apache, PHP, Perl, XML, Java, MySQL, > WebDAV, and more! http://www.apachecon.com/ > _______________________________________________ > Gusdev-gusdev mailing list > Gus...@li... > https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev |
|
From: Arnaud K. <ax...@sa...> - 2003-12-02 18:14:48
|
Steve Fischer wrote: > arnaud- > > i should have been more precise about DoTS::GeneInstanceCategory. We > have one row for it: "unknown". > > In general, it seems that our our CVs are either: > - loaded from an external source (by a plugin) > - our own invention (manually created). > > For the latter case, I agree that CBIL might want to make those > available to the GUS project in the form of XML. If my recollection > is correct, PSU has a copy of our GUS data from about a year or so > ago. Is that right? We've got XML data files used to populate gusdev CV tables (ExternalDatabase, AlgorithmParamKey, AlgorithmParamKeyType, RNASequenceType and SequenceType). I don't know if they are the ones you're talking about. We've updated ExternalDatabase XML to be used with GUS3. I don't think we need to update the other ones. AFAIK we don't have Anatomy, DevelopmentStage or Disease but I don't think we need them at the moment. I can see another table, SRes.EnzymeClass that we will want to populate. How shall we do it ? cheers Arnaud > The basic question is whether our current GUS XML is powerful enough > to express, eg, the anatomy CV. If so, then we could write it to XML > and make it available. > > steve > > Arnaud Kerhornou wrote: > >> Hi Steve >> >> Steve Fischer wrote: >> >>> Arnaud- >>> >>> first, about DoTS::GeneInstanceCategory. we don't seem to be using >>> this table. its empty in our instance. >> >> >> >> We want to generate gene, RNA and protein entries and link them to >> corresponding gene, RNA and protein feature entries. This is done >> using the Gene, RNA and Protein Instance entries. >> There is a FK to a GeneInstanceCategory entry in GeneInstance and >> this can not be null. The same for RNAInstance and for >> ProteinInstance, I'm not sure what they are for. >> >>> >>> >>> second, about the other vocabs. there is no central resource >>> available. each is dealt with individually. for example, sres.taxon >>> has a plugin which loads data from ncbi. sres.anatomy exists >>> uniquely in our db and was created manually. >> >> >> >> I'm thinking of CVs such as Taxon with has its own plugin and that >> the data can be retrieve from NCBI. >> There are a few ones, GUS specific, that we could share, e.g. >> SRes::OntologyRelationshipType >> <http://www.gusdb.org/cgi-bin/schemaBrowser?db=CBILBLD&table=SRes::OntologyRelationshipType&path=SRes::OntologyRelationshipType> >> (AFAIR two entries "part_of" and "is_a"). They could be stored in XML >> format or as SQL statements. I'm thinking of others CVs, I'll prepare >> a list. >> >>> >>> so, to best answer your question we would need to know which CVs you >>> want to populate. >>> >>> steve >>> >>> Arnaud Kerhornou wrote: >>> >>>> Hi >>>> >>>> We would like to populate some of the GUS Controlled Vocabulary >>>> tables. I'm thinking of DoTS::GeneInstanceCategory, and of a bunch >>>> of SRes tables. >>>> Are these resources available in XML format or in SQL statements >>>> from gusdb website or from CVS ? >>>> >>>> cheers and happy thanksgiving !! >>>> Arnaud >>>> >>>> |
|
From: Joan M. <ma...@pc...> - 2003-12-02 03:20:11
|
Hi Arnaud, > > We want to generate gene, RNA and protein entries and link them to > corresponding gene, RNA and protein feature entries. This is done > using the Gene, RNA and Protein Instance entries. > There is a FK to a GeneInstanceCategory entry in GeneInstance and this > can not be null. The same for RNAInstance and for ProteinInstance, I'm > not sure what they are for. We originally created GeneInstanceCategory, RNAInstanceCategory and ProteinInstanceCategory to describe using a controlled vocabulary the "origin" of a particular instance at the central dogma level. For example, the origin of a particular RNAInstance (or RNASequence) might be "assembly" and another RNAInstance of that RNA might be "mRNA" (perhaps a NCBI RefSeq). We have no plans to populate these three controlled vocabularies right now so you may want to create their vocabularies tailored to your own use. Joan |
|
From: Arnaud K. <ax...@sa...> - 2003-12-01 18:43:48
|
I have one rectification to make, see below. On Mon, 1 Dec 2003, Arnaud Kerhornou wrote: > Hi Steve > > Steve Fischer wrote: > > > Arnaud- > > > > first, about DoTS::GeneInstanceCategory. we don't seem to be using > > this table. its empty in our instance. > > We want to generate gene, RNA and protein entries and link them to > corresponding gene, RNA and protein feature entries. This is done using > the Gene, RNA and Protein Instance entries. > There is a FK to a GeneInstanceCategory entry in GeneInstance and this > can not be null. The same for RNAInstance and for ProteinInstance, I'm > not sure what they are for. > > > > > > > second, about the other vocabs. there is no central resource > > available. each is dealt with individually. for example, sres.taxon > > has a plugin which loads data from ncbi. sres.anatomy exists uniquely > > in our db and was created manually. > > I'm thinking of CVs such as Taxon with has its own plugin and that the > data can be retrieve from NCBI. I meant I'm not thinking of CVs such as Taxon. > There are a few ones, GUS specific, that we could share, e.g. > SRes::OntologyRelationshipType > <http://www.gusdb.org/cgi-bin/schemaBrowser?db=CBILBLD&table=SRes::OntologyRelationshipType&path=SRes::OntologyRelationshipType> > (AFAIR two entries "part_of" and "is_a"). They could be stored in XML > format or as SQL statements. I'm thinking of others CVs, I'll prepare a > list. > > > > > so, to best answer your question we would need to know which CVs you > > want to populate. > > > > steve > > > > Arnaud Kerhornou wrote: > > > >> Hi > >> > >> We would like to populate some of the GUS Controlled Vocabulary > >> tables. I'm thinking of DoTS::GeneInstanceCategory, and of a bunch of > >> SRes tables. > >> Are these resources available in XML format or in SQL statements from > >> gusdb website or from CVS ? > >> > >> cheers and happy thanksgiving !! > >> Arnaud > >> > >> > |
|
From: Steve F. <sfi...@pc...> - 2003-12-01 18:40:52
|
arnaud- i should have been more precise about DoTS::GeneInstanceCategory. We have one row for it: "unknown". In general, it seems that our our CVs are either: - loaded from an external source (by a plugin) - our own invention (manually created). For the latter case, I agree that CBIL might want to make those available to the GUS project in the form of XML. If my recollection is correct, PSU has a copy of our GUS data from about a year or so ago. Is that right? The basic question is whether our current GUS XML is powerful enough to express, eg, the anatomy CV. If so, then we could write it to XML and make it available. steve Arnaud Kerhornou wrote: > Hi Steve > > Steve Fischer wrote: > >> Arnaud- >> >> first, about DoTS::GeneInstanceCategory. we don't seem to be using >> this table. its empty in our instance. > > > We want to generate gene, RNA and protein entries and link them to > corresponding gene, RNA and protein feature entries. This is done > using the Gene, RNA and Protein Instance entries. > There is a FK to a GeneInstanceCategory entry in GeneInstance and this > can not be null. The same for RNAInstance and for ProteinInstance, I'm > not sure what they are for. > >> >> >> second, about the other vocabs. there is no central resource >> available. each is dealt with individually. for example, sres.taxon >> has a plugin which loads data from ncbi. sres.anatomy exists >> uniquely in our db and was created manually. > > > I'm thinking of CVs such as Taxon with has its own plugin and that the > data can be retrieve from NCBI. > There are a few ones, GUS specific, that we could share, e.g. > SRes::OntologyRelationshipType > <http://www.gusdb.org/cgi-bin/schemaBrowser?db=CBILBLD&table=SRes::OntologyRelationshipType&path=SRes::OntologyRelationshipType> > (AFAIR two entries "part_of" and "is_a"). They could be stored in XML > format or as SQL statements. I'm thinking of others CVs, I'll prepare > a list. > >> >> so, to best answer your question we would need to know which CVs you >> want to populate. >> >> steve >> >> Arnaud Kerhornou wrote: >> >>> Hi >>> >>> We would like to populate some of the GUS Controlled Vocabulary >>> tables. I'm thinking of DoTS::GeneInstanceCategory, and of a bunch >>> of SRes tables. >>> Are these resources available in XML format or in SQL statements >>> from gusdb website or from CVS ? >>> >>> cheers and happy thanksgiving !! >>> Arnaud >>> >>> > |
30 messages has been excluded from this view by a project administrator.
