[GUSDEV] try using a resources pipeline

[Steve F. <sfi...@pc...>]

fabricio and all gus users-

the best way to run plugins is with a resources pipeline. attached=20
please find a sample resources.prop file and resources.xml file.

you use a resources XML file with the 'loadresources' command. the=20
purpose is to be able to start with an empty gus instance, as produced=20
by the gus installation, and then with one command (loadresources) to=20
fill the database up with all the datasets you need. (you don't need to=20
start with an empty gus, its just that the system is powerful enough to=20
be able to do so).

the resources pipeline works in conjunction with a "resources=20
repository." The repository lives in your file system (at a location you=20
specify). For each resource (eg, SequenceOntology) it stores:
- the files for the resource
- the version of the resource
- the wget arguments used to get the resource from its ftp site
- or, if the resource is gotten "manually", ie, not automatically from=20
an ftp site, the details of the contact info, etc

the resources repository acts as a resource cache. the first time you=20
request a resource from it, the repository gets the file(s) from the=20
external ftp site. when you request that same resource again (eg, SO=20
version 1.4), the respository just gives you the resource that it is=20
already holding.

here is what the resources pipeline does
1. initializes a resource repository, if you don't already have one
2. initializes a pipeline directory in which all the files for the=20
pipeline are stored.
3. if you are not already in GUS as a user (eg, Core.UserInfo), it=20
automatically adds you (this detail is coming later today)
4. for each resource that you specify in the resources XML file:
- inserts into the db the ExternalDatabase and ExternalDatabaseRelease=20
for that resource
- calls the resources repository to acquire the resource (the XML=20
provides the wget command to do so)
- places the resource in a downloads/ directory in the pipeline/ director=
y
- calls any pre-processing required to prepare the resource for use by=20
plugins (this is called "unpacking")
- automatically registers the plugin you will use to load the resource=20
(this detail is coming in a few days)
- calls the plugin to load the data
- sets a signal in the pipeline/ dir to state that this step has been=20
completed. this way, you can re-run the pipeline and resources that are=20
already loaded won't be loaded again; only those that are new will be

by using a pipeline like this, you have created a detailed re-runnable=20
and complete specification for how you load your data into gus. at CBIL,=20
this is the backbone of how we manage all our big projects. it requires=20
some additional work up front. but, once you have the resources.xml, you=20
can always easily recreate your database from scratch, and have a=20
permanant and accurate record of how the db was built.

once you have used the resources pipeline to load all the resources,=20
then you can use an analysis pipeline to do all the analysis. (how to do=20
that is not discussed here).

steve

Fabr=EDcio wrote:

>Ok Steve,
>
>I'm trying to load a genbank file. More specifically, this file:
>
>http://www.de9.ime.eb.br/~fabricio/tvivax.gb
>
>I will do the tests and then I will report to you my tries ok?
>
>Thanks very much,
>
>Fabr=EDcio.
>
>
>-----Mensagem original-----
>De: Steve Fischer [mailto:sfi...@pc...]=20
>Enviada em: quarta-feira, 14 de setembro de 2005 21:33
>Para: Fabr=EDcio
>Assunto: Re: RES: InsertSequenceFeatures
>
>fabricio-
>
>ok, get the latest GUS/Supported.
>
>the plugin has been tested enough that its ready for you. let me know=20
>what problems you have.
>
>what kind of data are you loading?
>
>steve
>
>Fabr=EDcio wrote:
>
> =20
>
>>Ok Steve,
>>
>>I recognize your tasks... I know how many busy you must be.
>>
>>Thanks again,
>>
>>Fabr=EDcio.=20
>>
>>-----Mensagem original-----
>>De: Steve Fischer [mailto:sfi...@pc...]=20
>>Enviada em: ter=E7a-feira, 13 de setembro de 2005 22:41
>>Para: Fabr=EDcio
>>Assunto: Re: InsertSequenceFeatures
>>
>>fabricio-
>>
>>we are in the active process of testing and debugging the plugin. i am=20
>>so sorry that you have to wait for this. i made much good progress in=20
>>the debugging, and, i would be doing more right now, but, today and muc=
h=20
>>of tomorrow we are moving our databases to our big new servers. so, i=20
>>won't be finished until a couple more days.
>>
>>the plugin is my highest priority (we desperately need it too!)
>>
>>i think you'll really like the plugin when i release it.
>>
>>sorry again!
>>
>>steve
>>
>>Fabr=EDcio wrote:
>>
>>=20
>>
>>   =20
>>
>>>Hello Steve,
>>>
>>>Some days ago, I tried to insert a genbank file to GUS using your=20
>>>plugin InsertSequenceFeatures.pm. For that moment I received an error=20
>>>saying this:
>>>
>>>Can't locate object method "geneNameIds" via package=20
>>>"GUS::Supported::Plugin::InsertSequenceFeatures" at=20
>>>$GUS_HOME/lib/perl/GUS/Supported/Plugin/InsertSequenceFeatures.pm line=
 920
>>>
>>>I saw in plugin that there is a method call to geneNameIds, but I=20
>>>didn=92t find this method definition as =93sub=94 inside=20
>>>InsertSequenceFeatures.pm.
>>>
>>>Is it a not yet implemented method? Or is it another kind of error?
>>>
>>>Thanks for your help,
>>>
>>>Fabr=EDcio
>>>
>>>  =20
>>>
>>>     =20
>>>
>>=20
>>
>>   =20
>>
>
>
> =20
>