gusdev-gusdev

[Gusdev-gusdev] New LoadNRDB & Consolidated GUS install package

[Ed R. <ed_...@be...>]

As many of you know, we have been doing quite a few GUS installs down here, and this has pushed me to try and simplify this process as much as possible.  I am now far enough along on a couple things to bring them up on the list.

First, installing NRDB the first time in GUS is a horribly painful process using the exisintg plugin and this pain seems to be needless since it is an empty database.  To this end, I have written a couple scripts and a batch process for Oracle SQLLoader which accomplishes in about an hour what takes a few weeks with the plugin.  However, to make this work, I have to reserve early rows in a number of SRES tables for meaningful entries in columns such as row_alg_invocation_id.  Hence, my first discussion item:  Should we consider reserving early values in a number of the SRes tables to serve as standard values.  We already require that some rows be entered in GUS early on to make some-things work such as LoadReviewType.  It would seem that we should Pre-populate some of these tables with basic values that we can then refer to as standard values for bootstrapping operations such as a bulk load of NRDB.  Does anyone else see any value in this and, if so, what fileds should we create standard entries for?  Also, is there anything else that would be amenable to a batch process for bootstrapping?  (Note: I do NOT think any organisim specific data is amenable to bootstrapping.  That is what a (object based) pipeline is for.  Also, this batch process is only good if you are using Oracle, but a similar process cab be written there too.)

This also gets me to some of the other scripts we use to bootstrap GUS, such as the predefined set of ExternalDatabases we load.  The XML which I use to load this is pretty messy, and not well documented.  Does anyone mind if I clean it up?  If the answer is yes, is there anything I should know about this file?  it seems that the XML for this table load is a nice one to clean-up and make standard for GUS installations all over since it will push gus to be standardized across installations.  What else should we standardize?

Which now brings me to the last item I want to open up which is that I am close to completing a full GUS installation wrapper script which essentially makes a GUS installation a click-and-play operation.   One of our deliverables is supposed to be an easy to install GUS package.  Regardless of the state of GUS with regards to an official release, this script is going to make my life a whole lot easier.  I figure it might be nice to package the whole kit-n-kaboodle up into one nice fat tarball with a simple set of instructions for download from someplace.  Is anyone else interested in this?

Finally, one quick question I have about the NRDB load is that working on it showed me that the description filed in AASequenceIMP is too short for many of the descriptions in NRDB.  Do we want to up the description field size for dots.aasequenceimp?

Anyway, any feedback on this would be appreciated.

-Ed R


Ed Robinson
255 Deerfield Rd
Bogart, GA 30622
(706)425-9181

--Learn more about the face of your neighbor, and less about your own.
         -Sargent Shriver

Re: [Gusdev-gusdev] New LoadNRDB & Consolidated GUS install package

[Paul M. <pj...@sa...>]

On 15 Oct 2004, at 20:41, Ed Robinson wrote:

> This also gets me to some of the other scripts we use to bootstrap 
> GUS, such as the predefined set of ExternalDatabases we load.  The XML 
> which I use to load this is pretty messy, and not well documented.  
> Does anyone mind if I clean it up?  If the answer is yes, is there 
> anything I should know about this file?  it seems that the XML for 
> this table load is a nice one to clean-up and make standard for GUS 
> installations all over since it will push gus to be standardized 
> across installations.  What else should we standardize?
>

I documented and included the XML we required at Sanger to set-up a new 
instance of GUS;

http://www.gusdb.org/wiki/index.php/Bootstrap%20data
http://www.gusdb.org/wiki/index.php/Bootstrap%20data%20II

The first link is probably the common stuff that can be shared between 
sites. The ExternalDatabase XML is based on some XML we originally got 
from CBIL and I just added DBs as required so it probably needs to be 
cleaned up.
The second is more Sanger specific but it gives people XML they can at 
least edit and understand how someone else has implemented it.

Paul.

Re: [Gusdev-gusdev] New LoadNRDB & Consolidated GUS install package

[Michael S. <msa...@pc...>]

Ed,

Ed Robinson wrote:
> 
> Which now brings me to the last item I want to open up which is that
> I am close to completing a full GUS installation wrapper script which
> essentially makes a GUS installation a click-and-play operation.
> One of our deliverables is supposed to be an easy to install GUS
> package.  Regardless of the state of GUS with regards to an official
> release, this script is going to make my life a whole lot easier.  I
> figure it might be nice to package the whole kit-n-kaboodle up into
> one nice fat tarball with a simple set of instructions for download
> from someplace.  Is anyone else interested in this?
> 

As previously mentioned on this list, there is an effort underway to 
improve the administration of GUS, including a new turn-key installation 
mechanism (for both Oracle and Postgres).  It's my hope to have this 
implemented by the next GUS release (within the couple of months).

I think it's great that you have a wrapper script, which is probably of 
interest to several groups on this list, but I would warn against 
duplication of effort and encourage you to sign up and participate in 
this effort:

https://mail.pcbi.upenn.edu/mailman/listinfo/gusdba
http://mail.pcbi.upenn.edu/pipermail/gusdba/ (Archives)

Perhaps you could send your script to that list-- I think it will serve 
as a really good foundation for our installer efforts.

Thanks,

Mike

[Gusdev-gusdev] Pipeline Manager change

[Dave B. <db...@pc...>]

Hey all,

I noticed something that is a bit of a pain in the neck when running our 
pipeline; whenever we load external data, we have to make sure that there 
is a new entry in the ExternalDatabaseRelease table for that data.  The 
way I've always handled this is to make those by hand before the pipeline 
runs and set the database release IDs in the pipeline properties file.

I think a better way would be to have a step in the Pipeline that does it 
for you for each database release you have to load.  Values for the 
database release entry could either go in the properties file or be parsed 
from the file you are loading depending on availability.

This is easy enough to implement, but often we need to use the database 
release ids later in the pipeline.  There isn't any way to automatically 
set these as internal properties; we really need to add them by hand to 
the pipeline properties file.

I am going to add a method to Manager.pm that the pipeline can call named 
"waitForUser" (or something similar).  It will just exit the pipeline with 
a message, and in this case it can say "Please set the following property 
in the pipeline properties file:  flyDB=XXXX".  It is sort of a generic 
implementation of the 'exitToLiniac' method that we have already.  Then 
the user can set this property and start the pipeline again.

Dave

Re: [Gusdev-gusdev] Pipeline Manager change

[Steve F. <sfi...@pc...>]

Dave-

ok, but, the good news is that we will be renovating this department, 
and have in mind an improved facility for loading 3rd party data.

steve

Dave Barkan wrote:

> Hey all,
>
> I noticed something that is a bit of a pain in the neck when running 
> our pipeline; whenever we load external data, we have to make sure 
> that there is a new entry in the ExternalDatabaseRelease table for 
> that data.  The way I've always handled this is to make those by hand 
> before the pipeline runs and set the database release IDs in the 
> pipeline properties file.
>
> I think a better way would be to have a step in the Pipeline that does 
> it for you for each database release you have to load.  Values for the 
> database release entry could either go in the properties file or be 
> parsed from the file you are loading depending on availability.
>
> This is easy enough to implement, but often we need to use the 
> database release ids later in the pipeline.  There isn't any way to 
> automatically set these as internal properties; we really need to add 
> them by hand to the pipeline properties file.
>
> I am going to add a method to Manager.pm that the pipeline can call 
> named "waitForUser" (or something similar).  It will just exit the 
> pipeline with a message, and in this case it can say "Please set the 
> following property in the pipeline properties file:  flyDB=XXXX".  It 
> is sort of a generic implementation of the 'exitToLiniac' method that 
> we have already.  Then the user can set this property and start the 
> pipeline again.
>
> Dave
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out 
> more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev

[Gusdev-gusdev] Adding features to the sequence

[Jeetendra S. <so...@vb...>]

Hi all,

When we try to upload an existing sequence (and features) using the
GBParser, it updates the features of that sequence, i.e. it removes the
old and inserts the new ones.

However, is there a way to just ADD features (of any type) to the existing
sequence in the GUS DB ?

Thanks a lot for your help,
Jeetendra.

Re: [Gusdev-gusdev] Adding features to the sequence

[Steve F. <sfi...@pc...>]

Jeetendra-

I just talked to Angel who wrote this parser a long time ago.

He says that the parser is supposed to:
  - match the existing features in the db against the new ones mentioned 
in the gb file
  - if a feature is in the file but not in the db, it adds the feature
  - if the feature is in both, it leaves the feature alone
  - if the feature is in the db but not in the file, it deletes it.

is this not what you are observing?

steve

Jeetendra Soneja wrote:

>Hi all,
>
>When we try to upload an existing sequence (and features) using the
>GBParser, it updates the features of that sequence, i.e. it removes the
>old and inserts the new ones.
>
>However, is there a way to just ADD features (of any type) to the existing
>sequence in the GUS DB ?
>
>Thanks a lot for your help,
>Jeetendra.
>
>
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>Use IT products in your business? Tell us what you think of them. Give us
>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>http://productguide.itmanagersjournal.com/guidepromo.tmpl
>_______________________________________________
>Gusdev-gusdev mailing list
>Gus...@li...
>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>  
>

Re: [Gusdev-gusdev] Adding features to the sequence

[Jeetendra S. <so...@vb...>]

Steve,

     I'm referring to the last case in the list that Angel mentioned. It
deletes the feature if it is in Db but not in the file. We have a
scenario where new features are predicted and uploaded in different
steps. Thus, while uploading the new features predicted in later
steps, we don't want the features uploaded during previous steps to
be deleted.

  --Jeetendra.



> Jeetendra-
>
> I just talked to Angel who wrote this parser a long time ago.
>
> He says that the parser is supposed to:
>   - match the existing features in the db against the new ones mentioned
> in the gb file
>   - if a feature is in the file but not in the db, it adds the feature
>   - if the feature is in both, it leaves the feature alone
>   - if the feature is in the db but not in the file, it deletes it.
>
> is this not what you are observing?
>
> steve
>
> Jeetendra Soneja wrote:
>
>>Hi all,
>>
>>When we try to upload an existing sequence (and features) using the
>>GBParser, it updates the features of that sequence, i.e. it removes the
>>old and inserts the new ones.
>>
>>However, is there a way to just ADD features (of any type) to the
>> existing
>>sequence in the GUS DB ?
>>
>>Thanks a lot for your help,
>>Jeetendra.
>>
>>
>>
>>
>>-------------------------------------------------------
>>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>>Use IT products in your business? Tell us what you think of them. Give us
>>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
>> more
>>http://productguide.itmanagersjournal.com/guidepromo.tmpl
>>_______________________________________________
>>Gusdev-gusdev mailing list
>>Gus...@li...
>>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>>
>>
>


-- 
Jeetendra Soneja
Research Associate
Virginia Bioinformatics Institute
1880 Pratt Dr., Building XV
Blacksburg, VA 24060, USA
Phone: (540)-231-2789
http://www.vbi.vt.edu

Re: [Gusdev-gusdev] Adding features to the sequence

[Angel P. <an...@pc...>]

Jeetendra Soneja wrote:

>Steve,
>
>     I'm referring to the last case in the list that Angel mentioned. It
>deletes the feature if it is in Db but not in the file. We have a
>scenario where new features are predicted and uploaded in different
>steps. Thus, while uploading the new features predicted in later
>steps, we don't want the features uploaded during previous steps to
>be deleted.
>
>  --Jeetendra.
>
>  
>
Not that hard to fix. Add this line (assuming $flag is your new option) 
to line 1106

next if ($flag);


This will skip the deletion step in the if {} else{}  block
-angel

>
>  
>
>>Jeetendra-
>>
>>I just talked to Angel who wrote this parser a long time ago.
>>
>>He says that the parser is supposed to:
>>  - match the existing features in the db against the new ones mentioned
>>in the gb file
>>  - if a feature is in the file but not in the db, it adds the feature
>>  - if the feature is in both, it leaves the feature alone
>>  - if the feature is in the db but not in the file, it deletes it.
>>
>>is this not what you are observing?
>>
>>steve
>>
>>Jeetendra Soneja wrote:
>>
>>    
>>
>>>Hi all,
>>>
>>>When we try to upload an existing sequence (and features) using the
>>>GBParser, it updates the features of that sequence, i.e. it removes the
>>>old and inserts the new ones.
>>>
>>>However, is there a way to just ADD features (of any type) to the
>>>existing
>>>sequence in the GUS DB ?
>>>
>>>Thanks a lot for your help,
>>>Jeetendra.
>>>
>>>
>>>
>>>
>>>-------------------------------------------------------
>>>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>>>Use IT products in your business? Tell us what you think of them. Give us
>>>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
>>>more
>>>http://productguide.itmanagersjournal.com/guidepromo.tmpl
>>>_______________________________________________
>>>Gusdev-gusdev mailing list
>>>Gus...@li...
>>>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>>>
>>>
>>>      
>>>
>
>
>  
>

Re: [Gusdev-gusdev] New LoadNRDB & Consolidated GUS install package

[<pi...@pc...>]

I recently (within the last 3 weeks) loaded an entirely new version of nrdb and
it took less than 24 hours. This should have been equivalent to a first load. I
think that something else was wrong when you ran the plugin, possibly with the
database (e.g. indexes missing, a need to update statistics).

I agree that LoadNRDB needs an upgrade but I think its poor performance in this
case is due to some other problem.

                                                    -Debbie



Quoting Ed Robinson <ed_...@be...>:

> As many of you know, we have been doing quite a few GUS installs down here,
> and this has pushed me to try and simplify this process as much as possible.
> I am now far enough along on a couple things to bring them up on the list.
>
> First, installing NRDB the first time in GUS is a horribly painful process
> using the exisintg plugin and this pain seems to be needless since it is an
> empty database.  To this end, I have written a couple scripts and a batch
> process for Oracle SQLLoader which accomplishes in about an hour what takes a
> few weeks with the plugin.  However, to make this work, I have to reserve
> early rows in a number of SRES tables for meaningful entries in columns such
> as row_alg_invocation_id.  Hence, my first discussion item:  Should we
> consider reserving early values in a number of the SRes tables to serve as
> standard values.  We already require that some rows be entered in GUS early
> on to make some-things work such as LoadReviewType.  It would seem that we
> should Pre-populate some of these tables with basic values that we can then
> refer to as standard values for bootstrapping operations such as a bulk load
> of NRDB.  Does anyone else see any value in this and, if so, what fileds
> should we create standard entries for?  Also, is there anything else that
> would be amenable to a batch process for bootstrapping?  (Note: I do NOT
> think any organisim specific data is amenable to bootstrapping.  That is what
> a (object based) pipeline is for.  Also, this batch process is only good if
> you are using Oracle, but a similar process cab be written there too.)
>
> This also gets me to some of the other scripts we use to bootstrap GUS, such
> as the predefined set of ExternalDatabases we load.  The XML which I use to
> load this is pretty messy, and not well documented.  Does anyone mind if I
> clean it up?  If the answer is yes, is there anything I should know about
> this file?  it seems that the XML for this table load is a nice one to
> clean-up and make standard for GUS installations all over since it will push
> gus to be standardized across installations.  What else should we
> standardize?
>
> Which now brings me to the last item I want to open up which is that I am
> close to completing a full GUS installation wrapper script which essentially
> makes a GUS installation a click-and-play operation.   One of our
> deliverables is supposed to be an easy to install GUS package.  Regardless of
> the state of GUS with regards to an official release, this script is going to
> make my life a whole lot easier.  I figure it might be nice to package the
> whole kit-n-kaboodle up into one nice fat tarball with a simple set of
> instructions for download from someplace.  Is anyone else interested in this?
>
> Finally, one quick question I have about the NRDB load is that working on it
> showed me that the description filed in AASequenceIMP is too short for many
> of the descriptions in NRDB.  Do we want to up the description field size for
> dots.aasequenceimp?
>
> Anyway, any feedback on this would be appreciated.
>
> -Ed R
>
>
> Ed Robinson
> 255 Deerfield Rd
> Bogart, GA 30622
> (706)425-9181
>
> --Learn more about the face of your neighbor, and less about your own.
>          -Sargent Shriver
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>

Re: [Gusdev-gusdev] Pipeline Manager change

[John I. <io...@pc...>]

Adding a pipeline step for new external database release IDs is a great
idea.  I have noticed in the past that when people insert a record
"manually" (that is, with UpdateGusFromXML or SubmitRow), they tend to
supply only the minimum info, to the extent of populating required
fields with the string of "unknown".  I've been guilty of this myself. 
Hopefully, records created programmatically will document the DB release
better.

Question: is there a way to avoid editing the properties file to include
the new release ID?  

In many cases, we want the latest release of the given external
database.  Could we create a utility that takes an external database ID
and finds the ID of its latest release?

If we could do something like that, it would not only simplify a manual
step, it would completely automate it.

On Mon, 2004-10-25 at 17:39, Dave Barkan wrote:
> Hey all,
> 
> I noticed something that is a bit of a pain in the neck when running our 
> pipeline; whenever we load external data, we have to make sure that there 
> is a new entry in the ExternalDatabaseRelease table for that data.  The 
> way I've always handled this is to make those by hand before the pipeline 
> runs and set the database release IDs in the pipeline properties file.
> 
> I think a better way would be to have a step in the Pipeline that does it 
> for you for each database release you have to load.  Values for the 
> database release entry could either go in the properties file or be parsed 
> from the file you are loading depending on availability.
> 
> This is easy enough to implement, but often we need to use the database 
> release ids later in the pipeline.  There isn't any way to automatically 
> set these as internal properties; we really need to add them by hand to 
> the pipeline properties file.
> 
> I am going to add a method to Manager.pm that the pipeline can call named 
> "waitForUser" (or something similar).  It will just exit the pipeline with 
> a message, and in this case it can say "Please set the following property 
> in the pipeline properties file:  flyDB=XXXX".  It is sort of a generic 
> implementation of the 'exitToLiniac' method that we have already.  Then 
> the user can set this property and start the pipeline again.
> 
> Dave
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev

Re: [Gusdev-gusdev] Pipeline Manager change

[Dave B. <db...@pc...>]

Hey John--see below.

On Tue, 26 Oct 2004, John Iodice wrote:

> Adding a pipeline step for new external database release IDs is a great
> idea.  I have noticed in the past that when people insert a record
> "manually" (that is, with UpdateGusFromXML or SubmitRow), they tend to
> supply only the minimum info, to the extent of populating required
> fields with the string of "unknown".  I've been guilty of this myself.
> Hopefully, records created programmatically will document the DB release
> better.

Me too, but what I had in mind wasn't explicit functionality to load new 
DB releases, just a generic step in the Manager that exits and prints out 
information to the user.  I do remember writing a simple plugin that loads 
a new release into ExternalDatabaseRelease; it is in Common but I don't 
think it adds much over SubmitRow except to name ExternalDatabaseRelease 
attributes explicitly as parameters.

>
> Question: is there a way to avoid editing the properties file to include
> the new release ID?
>
> In many cases, we want the latest release of the given external
> database.  Could we create a utility that takes an external database ID
> and finds the ID of its latest release?

The only thing I could think of is to query for the latest release for a 
particular database, but this is dangerous; there could well be new 
releases loaded that for one reason or another we don't want to use (I 
think that is the case with GO right now; there is a new release loaded 
since the last time we made rules against GO terms.)  Maybe the latest 
release could be the default though, unless the user overrides it with a 
property.

I'd be interested to hear Steve's plans for loading 3rd party data and 
whether they include handling entries in the XDBR table.

Dave



>
> If we could do something like that, it would not only simplify a manual
> step, it would completely automate it.
>
> On Mon, 2004-10-25 at 17:39, Dave Barkan wrote:
>> Hey all,
>>
>> I noticed something that is a bit of a pain in the neck when running our
>> pipeline; whenever we load external data, we have to make sure that there
>> is a new entry in the ExternalDatabaseRelease table for that data.  The
>> way I've always handled this is to make those by hand before the pipeline
>> runs and set the database release IDs in the pipeline properties file.
>>
>> I think a better way would be to have a step in the Pipeline that does it
>> for you for each database release you have to load.  Values for the
>> database release entry could either go in the properties file or be parsed
>> from the file you are loading depending on availability.
>>
>> This is easy enough to implement, but often we need to use the database
>> release ids later in the pipeline.  There isn't any way to automatically
>> set these as internal properties; we really need to add them by hand to
>> the pipeline properties file.
>>
>> I am going to add a method to Manager.pm that the pipeline can call named
>> "waitForUser" (or something similar).  It will just exit the pipeline with
>> a message, and in this case it can say "Please set the following property
>> in the pipeline properties file:  flyDB=XXXX".  It is sort of a generic
>> implementation of the 'exitToLiniac' method that we have already.  Then
>> the user can set this property and start the pipeline again.
>>
>> Dave
>>
>>
>> -------------------------------------------------------
>> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>> Use IT products in your business? Tell us what you think of them. Give us
>> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>> http://productguide.itmanagersjournal.com/guidepromo.tmpl
>> _______________________________________________
>> Gusdev-gusdev mailing list
>> Gus...@li...
>> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>

Re: [Gusdev-gusdev] Pipeline Manager change

[John I. <io...@pc...>]

Dave,

That's a good point -- suppose Debbie is partway through the mouse build
and I start ApiDots; she wouldn't want to switch Prodom releases in
mid-build.

Would it be possible for the "get new release" stage to store the
release ID it created somewhere for later access, such as in a file
whose name was specified in the config file?

John


On Tue, 2004-10-26 at 11:42, Dave Barkan wrote:
> Hey John--see below.
> 
> On Tue, 26 Oct 2004, John Iodice wrote:
> 
> > Adding a pipeline step for new external database release IDs is a great
> > idea.  I have noticed in the past that when people insert a record
> > "manually" (that is, with UpdateGusFromXML or SubmitRow), they tend to
> > supply only the minimum info, to the extent of populating required
> > fields with the string of "unknown".  I've been guilty of this myself.
> > Hopefully, records created programmatically will document the DB release
> > better.
> 
> Me too, but what I had in mind wasn't explicit functionality to load new 
> DB releases, just a generic step in the Manager that exits and prints out 
> information to the user.  I do remember writing a simple plugin that loads 
> a new release into ExternalDatabaseRelease; it is in Common but I don't 
> think it adds much over SubmitRow except to name ExternalDatabaseRelease 
> attributes explicitly as parameters.
> 
> >
> > Question: is there a way to avoid editing the properties file to include
> > the new release ID?
> >
> > In many cases, we want the latest release of the given external
> > database.  Could we create a utility that takes an external database ID
> > and finds the ID of its latest release?
> 
> The only thing I could think of is to query for the latest release for a 
> particular database, but this is dangerous; there could well be new 
> releases loaded that for one reason or another we don't want to use (I 
> think that is the case with GO right now; there is a new release loaded 
> since the last time we made rules against GO terms.)  Maybe the latest 
> release could be the default though, unless the user overrides it with a 
> property.
> 
> I'd be interested to hear Steve's plans for loading 3rd party data and 
> whether they include handling entries in the XDBR table.
> 
> Dave
> 
> 
> 
> >
> > If we could do something like that, it would not only simplify a manual
> > step, it would completely automate it.
> >
> > On Mon, 2004-10-25 at 17:39, Dave Barkan wrote:
> >> Hey all,
> >>
> >> I noticed something that is a bit of a pain in the neck when running our
> >> pipeline; whenever we load external data, we have to make sure that there
> >> is a new entry in the ExternalDatabaseRelease table for that data.  The
> >> way I've always handled this is to make those by hand before the pipeline
> >> runs and set the database release IDs in the pipeline properties file.
> >>
> >> I think a better way would be to have a step in the Pipeline that does it
> >> for you for each database release you have to load.  Values for the
> >> database release entry could either go in the properties file or be parsed
> >> from the file you are loading depending on availability.
> >>
> >> This is easy enough to implement, but often we need to use the database
> >> release ids later in the pipeline.  There isn't any way to automatically
> >> set these as internal properties; we really need to add them by hand to
> >> the pipeline properties file.
> >>
> >> I am going to add a method to Manager.pm that the pipeline can call named
> >> "waitForUser" (or something similar).  It will just exit the pipeline with
> >> a message, and in this case it can say "Please set the following property
> >> in the pipeline properties file:  flyDB=XXXX".  It is sort of a generic
> >> implementation of the 'exitToLiniac' method that we have already.  Then
> >> the user can set this property and start the pipeline again.
> >>
> >> Dave
> >>
> >>
> >> -------------------------------------------------------
> >> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> >> Use IT products in your business? Tell us what you think of them. Give us
> >> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> >> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> >> _______________________________________________
> >> Gusdev-gusdev mailing list
> >> Gus...@li...
> >> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
> >
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > Use IT products in your business? Tell us what you think of them. Give us
> > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Gusdev-gusdev mailing list
> > Gus...@li...
> > https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
> >
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev

Re: [Gusdev-gusdev] Pipeline Manager change

[Dave B. <db...@pc...>]

> Would it be possible for the "get new release" stage to store the
> release ID it created somewhere for later access, such as in a file
> whose name was specified in the config file?

Hmm, maybe, and then the pipeline could just read in the release file when 
it starts up?  That would definitely be one way to do it.  However, then 
you essentially have two properties files, because I'm sure there will be 
times when you know which release you want to use and would have to set it 
in the database release file yourself.  So it is a trade-off between that 
and having the pipeline stop and tell the user to manually update the 
properties file when it creates a new release.  Both require minimum 
effort, but since these come up a lot, your idea is probably worth looking 
into.

Dave

>
> John
>
>
> On Tue, 2004-10-26 at 11:42, Dave Barkan wrote:
>> Hey John--see below.
>>
>> On Tue, 26 Oct 2004, John Iodice wrote:
>>
>>> Adding a pipeline step for new external database release IDs is a great
>>> idea.  I have noticed in the past that when people insert a record
>>> "manually" (that is, with UpdateGusFromXML or SubmitRow), they tend to
>>> supply only the minimum info, to the extent of populating required
>>> fields with the string of "unknown".  I've been guilty of this myself.
>>> Hopefully, records created programmatically will document the DB release
>>> better.
>>
>> Me too, but what I had in mind wasn't explicit functionality to load new
>> DB releases, just a generic step in the Manager that exits and prints out
>> information to the user.  I do remember writing a simple plugin that loads
>> a new release into ExternalDatabaseRelease; it is in Common but I don't
>> think it adds much over SubmitRow except to name ExternalDatabaseRelease
>> attributes explicitly as parameters.
>>
>>>
>>> Question: is there a way to avoid editing the properties file to include
>>> the new release ID?
>>>
>>> In many cases, we want the latest release of the given external
>>> database.  Could we create a utility that takes an external database ID
>>> and finds the ID of its latest release?
>>
>> The only thing I could think of is to query for the latest release for a
>> particular database, but this is dangerous; there could well be new
>> releases loaded that for one reason or another we don't want to use (I
>> think that is the case with GO right now; there is a new release loaded
>> since the last time we made rules against GO terms.)  Maybe the latest
>> release could be the default though, unless the user overrides it with a
>> property.
>>
>> I'd be interested to hear Steve's plans for loading 3rd party data and
>> whether they include handling entries in the XDBR table.
>>
>> Dave
>>
>>
>>
>>>
>>> If we could do something like that, it would not only simplify a manual
>>> step, it would completely automate it.
>>>
>>> On Mon, 2004-10-25 at 17:39, Dave Barkan wrote:
>>>> Hey all,
>>>>
>>>> I noticed something that is a bit of a pain in the neck when running our
>>>> pipeline; whenever we load external data, we have to make sure that there
>>>> is a new entry in the ExternalDatabaseRelease table for that data.  The
>>>> way I've always handled this is to make those by hand before the pipeline
>>>> runs and set the database release IDs in the pipeline properties file.
>>>>
>>>> I think a better way would be to have a step in the Pipeline that does it
>>>> for you for each database release you have to load.  Values for the
>>>> database release entry could either go in the properties file or be parsed
>>>> from the file you are loading depending on availability.
>>>>
>>>> This is easy enough to implement, but often we need to use the database
>>>> release ids later in the pipeline.  There isn't any way to automatically
>>>> set these as internal properties; we really need to add them by hand to
>>>> the pipeline properties file.
>>>>
>>>> I am going to add a method to Manager.pm that the pipeline can call named
>>>> "waitForUser" (or something similar).  It will just exit the pipeline with
>>>> a message, and in this case it can say "Please set the following property
>>>> in the pipeline properties file:  flyDB=XXXX".  It is sort of a generic
>>>> implementation of the 'exitToLiniac' method that we have already.  Then
>>>> the user can set this property and start the pipeline again.
>>>>
>>>> Dave
>>>>
>>>>
>>>> -------------------------------------------------------
>>>> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>>>> Use IT products in your business? Tell us what you think of them. Give us
>>>> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>>>> http://productguide.itmanagersjournal.com/guidepromo.tmpl
>>>> _______________________________________________
>>>> Gusdev-gusdev mailing list
>>>> Gus...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>>> Use IT products in your business? Tell us what you think of them. Give us
>>> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>>> http://productguide.itmanagersjournal.com/guidepromo.tmpl
>>> _______________________________________________
>>> Gusdev-gusdev mailing list
>>> Gus...@li...
>>> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>>>
>>
>>
>> -------------------------------------------------------
>> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>> Use IT products in your business? Tell us what you think of them. Give us
>> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>> http://productguide.itmanagersjournal.com/guidepromo.tmpl
>> _______________________________________________
>> Gusdev-gusdev mailing list
>> Gus...@li...
>> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>

[Gusdev-gusdev] speaking of pipelines

[Dave B. <db...@pc...>]

I was also thinking it would be nice to give pipelines another parameter 
(in addition to the properties file parameter they have now.)  The second 
one would be the --commit flag.  Right now committing plugins and scripts 
are done in the pipeline code itself, and the user has to change the code 
when 
they just want to test the pipeline and the plugins contained therein.

If we had a commit flag on the command line instead of in the code, then 
we could prevent having to change the code for testing, as well as force 
the user to explicitly state they want to commit code, which guards 
against accidentally running in commit mode.

How does that sound?  It would require some tweaking in the Manager 
(currently we have the explicit methods available, 'runPlugin' and 
'runPluginNoCommit' which would have to be changed to take the flag given 
on the command line), as well as in existing pipelines which are using the 
Manager, but it would be only a small amount of work.

Dave

[Gusdev-gusdev] Re: speaking of pipelines

[Dave B. <db...@pc...>]

Hey folks,

A follow-up on my email from last week suggesting adding a --commit flag 
to the pipeline API to handle committing plugins called by the pipeline.

Since I have heard two people say this is a good idea, and no one say it 
is a bad idea, I am going to make this change.

What this means for you is that after I put this into CVS (will do so 
soon), and you check it out, you will have to add the --commit flag to 
your command line in order to have any plugins in your pipeline run in 
commit mode.

I will make this backward compatible in pipeline code by having the method 
'runPluginNoCommit()' continue to run a plugin without committing the 
results, even if the --commit flag is set in the command line.  But I 
would suggest that you go through and change 'runPluginNoCommit' in your 
pipelines to 'runPlugin' and use the --commit flag only to 
restrict committing results.

I will let everyone know when this change is, um, committed to CVS.

Dave

On Tue, 26 Oct 2004, Dave Barkan wrote:

> I was also thinking it would be nice to give pipelines another parameter (in 
> addition to the properties file parameter they have now.)  The second one 
> would be the --commit flag.  Right now committing plugins and scripts are 
> done in the pipeline code itself, and the user has to change the code when 
> they just want to test the pipeline and the plugins contained therein.
>
> If we had a commit flag on the command line instead of in the code, then we 
> could prevent having to change the code for testing, as well as force the 
> user to explicitly state they want to commit code, which guards against 
> accidentally running in commit mode.
>
> How does that sound?  It would require some tweaking in the Manager 
> (currently we have the explicit methods available, 'runPlugin' and 
> 'runPluginNoCommit' which would have to be changed to take the flag given on 
> the command line), as well as in existing pipelines which are using the 
> Manager, but it would be only a small amount of work.
>
> Dave
>

Re: [Gusdev-gusdev] Adding features to the sequence

[Steve F. <sfi...@pc...>]

Jeetendra-

Did you get any help for this question yet?

steve

Jeetendra Soneja wrote:

>Hi all,
>
>When we try to upload an existing sequence (and features) using the
>GBParser, it updates the features of that sequence, i.e. it removes the
>old and inserts the new ones.
>
>However, is there a way to just ADD features (of any type) to the existing
>sequence in the GUS DB ?
>
>Thanks a lot for your help,
>Jeetendra.
>
>
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>Use IT products in your business? Tell us what you think of them. Give us
>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>http://productguide.itmanagersjournal.com/guidepromo.tmpl
>_______________________________________________
>Gusdev-gusdev mailing list
>Gus...@li...
>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>  
>

Re: [Gusdev-gusdev] Adding features to the sequence

[Jeetendra S. <so...@vb...>]

Hi Steve,
        Since this requirement would be mandatory for our project, I
decided to try and modify the plugin myself. I am trying to add a
command-line option --addFeatures, which would tell GBParser to
just add the features (rather than delete and reinsert). However,
I haven't had a chance to test it.
        I would appreciate your comments on the same.

Thank you,
Jeetendra.



> Jeetendra-
>
> Did you get any help for this question yet?
>
> steve
>
> Jeetendra Soneja wrote:
>
>>Hi all,
>>
>>When we try to upload an existing sequence (and features) using the
>>GBParser, it updates the features of that sequence, i.e. it removes the
>>old and inserts the new ones.
>>
>>However, is there a way to just ADD features (of any type) to the
>> existing
>>sequence in the GUS DB ?
>>
>>Thanks a lot for your help,
>>Jeetendra.
>>
>>
>>
>>
>>-------------------------------------------------------
>>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>>Use IT products in your business? Tell us what you think of them. Give us
>>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
>> more
>>http://productguide.itmanagersjournal.com/guidepromo.tmpl
>>_______________________________________________
>>Gusdev-gusdev mailing list
>>Gus...@li...
>>https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>>
>>
>


-- 
Jeetendra Soneja
Research Associate
Virginia Bioinformatics Institute
1880 Pratt Dr., Building XV
Blacksburg, VA 24060, USA
Phone: (540)-231-2789
http://www.vbi.vt.edu