genex-dev — GeneX developers

Re: [GeneX-dev] [Fwd: Re: about GeneX DB schema.]

[Jiaye Z. <ze...@in...>]

Hi Andrea,

Quoting Harry Mangalam <man...@ho...>:

> > Andrea Splendiani wrote:
> questions:
> 1) since I'm going to use fields probably different from yours, which is
> the
> minimus set of infos that are required to build a valid GeneXML document
> ?

The GeneXML DTD speficies the entities in GeneXML. Most of, if not all, elements 
and attributes are optional according to the DTD. So if one validates a 
lightweight GeneXML file using a validating parser, this should succeed as long 
as the referential integrety within GeneXML is satisfied. That being said, when 
the GeneXML file is parsed and data inserted into the database, the database may 
have separate requirements it requires. If you select the minimium 
GeneXML data needed and map it to EBI schema, as long as the GeneXML file is 
well formed and valid, and the data requirement by the DB is met, you are ok.

> By the way, here there's a lot of confusion abaut DB design, so that my
> idea is
> to develop something as quick as possible to have a prototype and to
> gain some
> hint from its usage. Then I may change the DB-schema (but I have to
> choose data
> now in a way that they'll be compatible to the new schema).

It's always good to have as much data as possible, one can always filter things 
out. But it would be hard to generate data when there is none. Seems sticking to 
a fairly generic, comprehensive data format will give you some flexibility 
later.

> Another question: all your tools have a GeneXML interarface ? 

That would be nice...:)

> What have you used to develop the GeneXML parser ? And to develop the
> "dumper"
> is there any tool or it is necessary to make it in a custom way ?

Currently we have XML2DB and DB2XML, which transfer data between the database 
and GeneXML. Both are written in Perl, using XML::DOM module, connection with 
database is done with DBI via an BIO::GeneX API (now available on CPAN). We are 
also in the process of writing a more generic GeneXML API that allows people to 
write applications using GeneXML.

Jiaye

[GeneX-dev] [Fwd: Re: about GeneX DB schema.]

[Harry M. <man...@ho...>]

-------- Original Message --------
Subject: Re: about GeneX DB schema.
Date: Wed, 1 Mar 2000 13:53:29 +0100
From: Andrea Splendiani <an...@bt...>
To: Harry Mangalam <man...@ho...>
References: <010...@lo...> <010...@bt...> <3A9...@ho...>

Il mer, 21 feb 2001, hai scritto:
> Hi Andrea - nice to hear from you again.  I'm taking the liberty of posting my reply to the NCGR GeneX develeopment list.  If you want to have even wider response, you could also post it to: the
> Sourceforge list which has even more readers who might care to respond.
> 
> Andrea Splendiani wrote:
> If I follow you, this shouldn't matter very much - Affy data is fine for our schema and you are allowed to have null data in many of the fields that desccribe the technology
I've tried to use your schema but I've seen that there's no agreement from the
biologists on the meaning of the fields in tables!
I'm now trying to start from EBI schema, which has an higher view. But I still
want the DB to have a Genex compatible output (i.e.: GeneXML).
Planning to build one day my own import/export filetr to GeneXML, I have to
questions:
1) since I'm going to use fields probably different from yours, which is the
minimus set of infos that are required to build a valid GeneXML document ?
2) In the Experiment description, if I fallow EBI schema, I'll have data
structered in samples and teatment between them... say
sample-delay-sample-delay for a cinetics. Do you structure it in the same way
? If not, how can I map it between the two way ?

By the way, here there's a lot of confusion abaut DB design, so that my idea is
to develop something as quick as possible to have a prototype and to gain some
hint from its usage. Then I may change the DB-schema (but I have to choose data
now in a way that they'll be compatible to the new schema).

Another question: all your tools have a GeneXML interarface ? Even flat-file or
some other format ?
What have you used to develop the GeneXML parser ? And to develop the "dumper"
is there any tool or it is necessary to make it in a custom way ?

bye,
Andrea Splendiani

[GeneX-dev] New approach for GeneX DataLoader

[Harry M. <man...@ho...>]

Hi All,

Jason, myself, and Michael Pear met yesterday to work on user issues
about dataloading and usability.

We had a near-religious experience / conversion over lunch in terms of 
how to address the Curation Tool and DataLoading in general & then 
continuing on for several hours.  
We're pretty unanimous that the Curation Tool as written
is not a viable candidate for large-scale data loading, and therefore not
particularly suited for data pipelines or large scale gene expression
projects. 

So we came up with a sketch of a new approach that simultaneously seems to
solve a lot of issues relating to time, confusion, user interaction, data
transport, scaling, security, allows both Mac & Win32 usability, uses only
Open Source Software, solves a good bit of the NSF-funded curation
problem,  and solves world hunger.

I'll get to the description below, but this new approach works because it
requires that we flip our approach to GeneX in terms of what it addresses
1st. We had initially designed GeneX as a National Resource DB thingie (NRDB), so
we approached it with the idea that labs would be stuffed with
well-characterized, validated data waiting for a home.  This perspective
(largely mine) is faulty.  Users are still in the mode where they need to
get their data analyzed in a way that is usable by mere humans - they need
a LAB database, not a NRDB - at least not yet..

Large scale Gene Expression data is also often done by core facilities
that use specific scanners & spotters, a restricted set of Array Layouts,
and sequence features.  Users really don't want to enter this data over
and over again.  The CT was going to support templates, but things that
are template-able tend to be scriptable as well.

The current thinking is to help users get to the point of loading,
analyzing, and publishing their data so that they become comfy with the
idea of releasing their data to a NRDB.  THEN we can start thinking about
implementing the infrastructure tools for such a beast.  The nice thing is
that we've really designed the DB infrastructure for this, so there's not
much to do on that front, but the curation tool DOES have to be
re-implemented.  

We think that the best way to do this is as a server-side tool.

The way it would work is as follows:

The user becomes less central in the actual data loading - he still has to
describe the data, tho not in as much detail as the CT requires -
basically 1-3 pages of web form and then typically submits the description
to the DB to get an ID.  That ID is used to tag the core numeric data
being uploaded to the DB by the user or facility (call it CORE) that
actually does the scanning and the initial data extraction.  The CORE
places the data files in a particular format generated by the experiment
on the GeneX server in a paricular directory and fills in a web form that
associates the experiment description ID tag with the data and notifies
the server to start a script that looks in that directory and
automatically parse the data files (they can be native excel files -
Michael found a perl Excel parser) and load it into the database by direct
connection thru DBI calls.  This code  is already mostly written by Jason
as he used it to load the initial data into the DB.

This now becomes almost entirely a server process & removes the client
application (the Curation Tool / MaxdLoad) from the equation, allows PC &
Mac compatibility (via samba and netatalk), allows native unix pipes and
transport protocols to be used to move data around even thru firewalls,
even removes the data manipulation from the users (unless they want to
have access to it). It also allows asynchronous bulk data loading (Michael
gave us the example of a postdoc who wants to load 300 arrays - would YOU
want to traverse the CT 300 times?  The login procedure to the unix server
enforces a measure of security and that can be used to pass a useful (?)
level of security to the DB.

Our initial focus (as MP has a larrge data set to handle) is not to solve
the universal/generic problem of data loading, but to specifically focus
on his immediate problems (output from QuantaArray and an array layout
from Biorobotics, and Molecular Dynamics data for the ~300 files.  We also
will be of course interested in handling our own problems as to how to
curate in the datassets we sais that we'd do for the NSF grant.

Jason has also added the foreign key constraints back into the DB with the
result that we should be able to add true 'cascade deletes ' of experiment
sets. Jason & Michael also made a stab at adding security fields to the
nec tables so that we can establish array-level security to GeneX.

We're going to start discussing this in more detail on the genex-dev list
(trying to keep discussion in one place until we get complaints),
prefixing DataLoader threads with 'DL:'

Lonnie and Todd, how do you feel about this approach?  It removes the
wrestling with the 'nest of anacondas' and delays the re-design for a
bit.  Admittedly a lot of the discussion will be perlish in nature, but
we'd really like you to contribute from your experience with the CT.

Bjarte & Inge, this doesn't particularly effect you or the
J-Express<->GeneX connection except that it will make mo' data / mo'
available / mo' faster / mo' bettah.

David, this takes a slightly different approach than your maxdLoader, but
the nice thing is that it also makes is somewhat easier (well, less
complex anway?. well, more perlish, and therefore more accessible to more
people (I'm pretty sure the bio-perl community is larger than the Bio-Java
community...?) to add support for more scanners, spotters, and prelim
analysis and should still allow you to load it to maxdSQL via a servlet or
even a slightly modified Genex.pm approach, depending on how different the
schemas are.. How do you feel about such an approach?
   It also addresses the bulk-loading aspect of larger systems, altho I
think you had started to address this in maxdLoad - ?

-- 
Cheers,
Harry

Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... || man...@ho...

Re: [GeneX-dev] Hybridization field in schema (2/8/01)

[Todd P. <tf...@nc...>]

I am actively converting the ERWin diagram to an opensource modeling tool
called DOME (http://www.htc.honeywell.com/dome). This will allow
modification to the model by somebody not sitting at our only machine with
an ERWin license. The DOME tool allows the developer to customize auto-code
or even the modeling environment. I have used an earlier (and more crude)
version of the tool to auto-generate CLOS class files from a Coad-Yourdon
class diagram (more complicated than our current diagram) and it worked
great.

Anyway, I still have a bit of work to do on the model and then will release
the new diagram and data. Thanks for everybody's patience.

As far as modification to the model for hyb name...the parameters are in the
XML files and are handled by the curation tool. The programs XML2DB and
DB2XML would need to be looked at (what are they doing now with hyb name?).

Todd

----- Original Message -----
From: "Jason E. Stewart" <ja...@op...>
To: <gen...@li...>
Sent: Monday, February 26, 2001 9:04 AM
Subject: Re: [GeneX-dev] Hybridization field in schema (2/8/01)


> "Michael Pear" <mic...@ho...> writes:
>
> > I looked at the latest schema available at
> > http://www.ncgr.org/research/genex/GeneXSchema.pdf and see that the
> > field "hybridization_name" no longer appears. This is an important
> > field as there is currently no otherway to tie array measurements
> > together other than in an experiment set (at least that I can
> > see). We plan on using it, for example, to identify array
> > measurements from the same glass slide using a bar code reference.
> >
> > Is this an error, or do I lose this field if I update to the latest
> > release on sourceforge?
>
> You will not lose anything (and there is no latest version on SF :-(
>
> That diagram is created by data from the ERWin program. Currently that
> program is not used to actually create the database (it cannot produce
> Postgres td scripts). Todd has been working to keep ERWin up-to-date
> with the state of the DB, but unfortunately I have not been very good
> at communicating changes, nor have I checked the diagram.
>
> I'm still working to get a new version of the updater available, but
> I've been busy at the OMG meeting.
>
> jas.
>
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> http://lists.sourceforge.net/lists/listinfo/genex-dev
>

Re: [GeneX-dev] Hybridization field in schema (2/8/01)

[<ja...@op...>]

"Michael Pear" <mic...@ho...> writes:

> I looked at the latest schema available at
> http://www.ncgr.org/research/genex/GeneXSchema.pdf and see that the
> field "hybridization_name" no longer appears. This is an important
> field as there is currently no otherway to tie array measurements
> together other than in an experiment set (at least that I can
> see). We plan on using it, for example, to identify array
> measurements from the same glass slide using a bar code reference.
> 
> Is this an error, or do I lose this field if I update to the latest
> release on sourceforge?

You will not lose anything (and there is no latest version on SF :-(

That diagram is created by data from the ERWin program. Currently that
program is not used to actually create the database (it cannot produce
Postgres td scripts). Todd has been working to keep ERWin up-to-date
with the state of the DB, but unfortunately I have not been very good
at communicating changes, nor have I checked the diagram.

I'm still working to get a new version of the updater available, but
I've been busy at the OMG meeting.

jas.

[GeneX-dev] Hybridization field in schema (2/8/01)

[Michael P. <mic...@ho...>]

I looked at the latest schema available at http://www.ncgr.org/research/genex/GeneXSchema.pdf
and see that the field "hybridization_name" no longer appears. This is an important
field as there is currently no otherway to tie array measurements together other than in
an experiment set (at least that I can see). We plan on using it, for example, to identify array
measurements from the same glass slide using a bar code reference.

Is this an error, or do I lose this field if I update to the latest release on sourceforge?

I also notice a missing relationship in the schema diagram...AL_Spots is connected
to ArrayLayout by virtue of the foreign key al_fk, but no connection exists in the diagram.

Regards,

Michael Pear

Re: [GeneX-dev] Use of Bio::Genex API

[<ja...@op...>]

"Michael Pear" <mic...@ho...> writes:

> While you were vacationing (and the rest of us were working, I might
> add:-), I've become more familiar with the API, really simplified
> many of the things I was trying.  

Hey, now I take that as a bit of *good* news, even a compliment?

> I understand you are coming our way, and I hope we can get
> together. So I can show you what I'm up to when you are here.

Yes. I'm arriving in LA Saturday evening, and I'll drive down to
Harry's place. I'll be at an OMG data format get-together Sun and Mon,
and then Tue I'll head in to the UCI Genomics group meeting. Maybe you
and Harry and I can all have dinner Sat/Sun/Mon?

> One note on the documentation is that "get_all_objects" isn't mentioned, and the
> "get_objects" with a list of keys doesn't work, as specified in the
> documentation.

Sorry. I broke the documenation when I split get_objects() into two
functions. The new function get_all_objects() returns all objects of a
given class. It can be restricted by specifying an anonymous hash with
'column' and 'value' keys:

@objs = Bio::Genex::UserSec->get_all_objects({column=>'login',
                                              value=>'test_user'});

get_objects() now *only* accepts a list of ids:

@objs = Bio::Genex::UserSec->get_objects(64,67,68);

I just gave this a test run in the debugger and it works for me using
'2.6.0'. Do you have any examples that are problematic?

  DB<3> use Bio::Genex::UserSec

  DB<4> x Bio::Genex::UserSec->get_all_objects
0  Bio::Genex::UserSec=SCALAR(0x10629df4)
   -> 0
1  Bio::Genex::UserSec=SCALAR(0x102eae94)
   -> 1
  DB<5> @a =  Bio::Genex::UserSec->get_all_objects

  DB<6> x $a[0]->us_pk
0  64
  DB<7> x $a[1]->us_pk
0  67
  DB<8> x Bio::Genex::UserSec->get_objects(64,67)
0  Bio::Genex::UserSec=SCALAR(0x10602f30)
   -> 2
1  Bio::Genex::UserSec=SCALAR(0x1069032c)
   -> 3
  DB<9> @b = Bio::Genex::UserSec->get_objects(64,67)

  DB<10> x $b[0]->us_pk
0  64
  DB<11> x $b[0]->con_fk
0  525
  DB<12> x $b[0]->con_obj->contact_person
0  'test_contact_person'
  DB<13> use Bio::Genex

  DB<14> x $Bio::Genex::VERSION
0  '2.6.0'


jas.

Re: [GeneX-dev] Use of Bio::Genex API

[Michael P. <mic...@ho...>]

Expecting an explicit use or require for each table is not a burden. It just
came up because of the circumstances of my first attempts.  I assumed that
 "use Bio::Genex;" was sufficient.

While you were vacationing (and the rest of us were working, I might add:-),
I've become more familiar with the API,  really simplified many of the things I was trying.
I understand you are coming our way, and I hope we can get together. So I can show 
you what I'm up to when you are here.

One note on the documentation is that "get_all_objects" isn't mentioned, and the
"get_objects" with a list of keys doesn't work, as specified in the documentation.

Michael Pear
----- Original Message ----- 
From: "Jason E. Stewart" <ja...@op...>
To: <gen...@li...>
Sent: Friday, February 23, 2001 7:46 AM
Subject: Re: [GeneX-dev] Use of Bio::Genex API


> "Michael Pear" <mic...@ho...> writes:
> 
> > Solved my problem. UserSec was apparently loaded
> > through "check_password" and available for use, but none
> > of the other table classes (like Contact) were.  So everything
> > worked with UserSec, but nothing else. Fixed by adding
> > a "require Bio::Genex::Contact;" before using this.
> 
> Hi Michael,
> 
> First, I'm back from Mexico. Fire away any more questions.
> 
> Second, it's really cool that you're now stress-testing the API, I'll
> have to practice my duck-and-cover techniques again ;-)
> 
> The issue of which modules to include (with a 'use' or 'require'
> pragma), should be fixed in the documentation (someplace??). 
> 
> The issue is that loading modules into memory takes time. Since there
> is a module for every table in the DB (which is a lot of modules) most
> applications will only need to access at most a few, so it would be a
> major unecessary slowdown to include them all by default. 
> 
> The guideline in Perl is that whenever you are going to access
> anything out of a class, either a variable or a class method, you have
> to 'use' that class. Calling ->new() is one such case, another is
> ->get_all_objects() and ->get_objects(). 
> 
> I am open to suggestions as to how to improve this.
> 
> jas.
> 
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> http://lists.sourceforge.net/lists/listinfo/genex-dev

Re: [GeneX-dev] Use of Bio::Genex API

[<ja...@op...>]

"Michael Pear" <mic...@ho...> writes:

> Solved my problem. UserSec was apparently loaded
> through "check_password" and available for use, but none
> of the other table classes (like Contact) were.  So everything
> worked with UserSec, but nothing else. Fixed by adding
> a "require Bio::Genex::Contact;" before using this.

Hi Michael,

First, I'm back from Mexico. Fire away any more questions.

Second, it's really cool that you're now stress-testing the API, I'll
have to practice my duck-and-cover techniques again ;-)

The issue of which modules to include (with a 'use' or 'require'
pragma), should be fixed in the documentation (someplace??). 

The issue is that loading modules into memory takes time. Since there
is a module for every table in the DB (which is a lot of modules) most
applications will only need to access at most a few, so it would be a
major unecessary slowdown to include them all by default. 

The guideline in Perl is that whenever you are going to access
anything out of a class, either a variable or a class method, you have
to 'use' that class. Calling ->new() is one such case, another is
->get_all_objects() and ->get_objects(). 

I am open to suggestions as to how to improve this.

jas.

Re: [GeneX-dev] Use of Bio::Genex API

[Michael P. <mic...@ho...>]

Solved my problem. UserSec was apparently loaded
through "check_password" and available for use, but none
of the other table classes (like Contact) were.  So everything
worked with UserSec, but nothing else. Fixed by adding
a "require Bio::Genex::Contact;" before using this.

M. Pear
> 
> ----- Original Message -----
> From: "Michael Pear" <mic...@ho...>
> To: "GeneX Dev List" <gen...@li...>
> Sent: Monday, February 12, 2001 8:47 AM
> Subject: [GeneX-dev] Use of Bio::Genex API
> 
> 
> > Hi Jason,
> >
> > I've started experimenting with the Bio::Genex API and ran into the
> following
> > problem with what looks like a pretty straightforward bit of code:
> >
> >     ....
> >     my $UserSec = new Bio::Genex::UserSec(id=>$userid);
> >     my $contact_key=$UserSec->con_fk();
> >     my $Contact = new Bio::Genex::Contact(id=>$contact_key);
> >     $udat{userName} = $Contact->contact_person();
> >     ....
> > But when I run this in my program I get:
> >
> >
> > [9836]ERR: 24: Line 18: Error in Perl code: Can't locate object method
> "new" via package
> > "Bio::Genex::Contact" at /var/genex/html/genex/mrp/genex-login.plhtml line
> 49.
> >
> > I've confirmed that $userid is set (from check_password) and if I comment
> the offending line, and
> > replace the
> > clal to contact_person() with a text string, my program works fine. I've
> also confirmed that
> > $contact_key is
> > set to an existing entry in the contact table.
> >
> > Ideas on what I should look for?
> >
> > Regards,
> >
> > Michael Pear
> >
> >
> > _______________________________________________
> > Genex-dev mailing list
> > Gen...@li...
> > http://lists.sourceforge.net/lists/listinfo/genex-dev
> >
> 
> 
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> http://lists.sourceforge.net/lists/listinfo/genex-dev

Re: [GeneX-dev] Use of Bio::Genex API

[Todd P. <tf...@nc...>]

Michael,

Everybody (just about) is in Albuquerque at UNM today since Harry came out
to visit. Jason is in Mexico for 2 weeks. Your problem looks like a method
dispatching problem...is there a method which matches the signature
(constructor of Contact with a single argument of integer)? I'm not a Perl
expert, but know a lot about OO techniques. In Java or C++, this error is
typical when arguments and types don't match exactly (C++ is really picky).

Hope this helps.

Todd

----- Original Message -----
From: "Michael Pear" <mic...@ho...>
To: "GeneX Dev List" <gen...@li...>
Sent: Monday, February 12, 2001 8:47 AM
Subject: [GeneX-dev] Use of Bio::Genex API


> Hi Jason,
>
> I've started experimenting with the Bio::Genex API and ran into the
following
> problem with what looks like a pretty straightforward bit of code:
>
>     ....
>     my $UserSec = new Bio::Genex::UserSec(id=>$userid);
>     my $contact_key=$UserSec->con_fk();
>     my $Contact = new Bio::Genex::Contact(id=>$contact_key);
>     $udat{userName} = $Contact->contact_person();
>     ....
> But when I run this in my program I get:
>
>
> [9836]ERR: 24: Line 18: Error in Perl code: Can't locate object method
"new" via package
> "Bio::Genex::Contact" at /var/genex/html/genex/mrp/genex-login.plhtml line
49.
>
> I've confirmed that $userid is set (from check_password) and if I comment
the offending line, and
> replace the
> clal to contact_person() with a text string, my program works fine. I've
also confirmed that
> $contact_key is
> set to an existing entry in the contact table.
>
> Ideas on what I should look for?
>
> Regards,
>
> Michael Pear
>
>
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> http://lists.sourceforge.net/lists/listinfo/genex-dev
>

[GeneX-dev] Use of Bio::Genex API

[Michael P. <mic...@ho...>]

Hi Jason,

I've started experimenting with the Bio::Genex API and ran into the following
problem with what looks like a pretty straightforward bit of code:

    ....
    my $UserSec = new Bio::Genex::UserSec(id=>$userid);
    my $contact_key=$UserSec->con_fk();
    my $Contact = new Bio::Genex::Contact(id=>$contact_key);
    $udat{userName} = $Contact->contact_person();
    ....
But when I run this in my program I get:


[9836]ERR: 24: Line 18: Error in Perl code: Can't locate object method "new" via package
"Bio::Genex::Contact" at /var/genex/html/genex/mrp/genex-login.plhtml line 49.

I've confirmed that $userid is set (from check_password) and if I comment the offending line, and
replace the
clal to contact_person() with a text string, my program works fine. I've also confirmed that
$contact_key is
set to an existing entry in the contact table.

Ideas on what I should look for?

Regards,

Michael Pear

[GeneX-dev] [Fwd: Re: Fwd: Re: Fwd: Re: infos]

[Harry M. <man...@ho...>]

This was sent to me directly and it contains enough ?s for everyone now :)

answer what you want.  I'll try to fill in the missing bits.


-------- Original Message --------
Subject: Re: Fwd: Re: Fwd: Re: infos
Date: Thu, 8 Feb 2001 13:49:56 +0100
From: Andrea Splendiani <an...@bt...>
To: man...@ho...
References: <010...@lo...>

> Subject: Re: Fwd: Re: infos
> Date: Mon, 05 Feb 2001 10:00:17 -0800
> From: Harry Mangalam <man...@ho...>
> To: ser...@ti...,
>  genexdev at SF <gen...@li...>
 
> > The problem is that if I have some information represented in the db in a
> > non-structured way (eg. experiments decribed in human language), and then I
> > have the structured in XML (eg: experimets is ID + #Arrays etc. tec.) I may
> > get into trouble.
> > >From what I've seen it may be a problem, since many fileds are left to
> > > "human description".
>  If I understand you, it won;t nec. be aproblem in representing the
>  unstructured text in XML - it can just be represented as TEXT, but as such,
>  if you want to be able to extract information from it in the DB, you'll have
>  to come up with a way of indexing it in SW - something like having a script
>  that  does a daily text-index of those fields so that they can also be
>  searchable   (but outside the DB system.  Most DB systems allow things like
>  regex   searches, but (I think) most such systems don't have contextual text
>  searches, such as a glimpse / isite search capability.
the problem I was wondering about is:
when I store data in the DB I want to store a description of the exsperiment
too (+ other meta-data). I may store it in a more or less structured way, and
there may be a trade-off between how much data is structured, and how flexible
is the DB.
Then the problem is to find the best trade-off so that a query has not to rely
on text search on descriptions made perhaps without even a very standard
glossary.
As an example see the EBI schema in which an experiments consisting of ten
samples separated by 10 minutes may in part be described in a structured way,
ten samples "entity" linked by a relation "treatment" with value "dalay". Then
this is easier to analyze then a text description in a filed "experiment" like
"the experiment consists of ten samples taken at 10 minutes intervals....".
Then if I have this semi-strucured view of an axperiment, I want to exchange it
in an XML document, or if, as an example, I receive a document in which the
experiment is a text description like "the experiments consists of...", how
can put it in my DB where such informations are more structured ?
I hope I've been clear, I fear I haven't!

Anyway, I'll see better the schemas proposed, the XML languages, and I'll see
how things are done and if such problems really exists.    

...
> > Why do you state your DB runs on PostgresSQL and not MySQL ? Do you use SQL
> > functions not provided by MySQL ? which are they ?
> > MySQL should be the fastest one, and it should have all the functionality
> > needed by a project like a DB for gene expression data (mainly storage, no
> > transactions, scarce writing etc.).
> We originally considered MySQL but decided for Postgres because of lack of
>  some consistency checks, constraints, and transactions in MySQL.  While I
>  think MySQL is a good system for a lab DB, if it has to scale quite large,
>  Postgres is a better choice - (At least it was - I've heard that MySQL now
>  supports some of the features that we needed - if we re-visit this, we may
>  add support for MySQL as well - if you want to try to implement GeneX on
>  MySQL, FANTASTIC!! - we certainly would like to support both, but there are
>  constraints as to number of people we can throw at it.
I'll start to work actively on the DB after the 20 of this month.
I'll probabl have a look at GeneX, but probably I will not fallow it entirely,
also becouse here they are quite specialized (like only Affimetrix, only
Murine, only Onchology...), and they may want an additional set of features for
their own functional annotation...
Anyway, I'll see, maybe I'll use GeneX, and in that case I'll surely try MySQL
with it. I'll let you know...


> > Thanks, I had a look at all that (brief by now, but I'll get back on it!)
> > Well, I have to set up a DB for microarray experiment data. Size will start
> > from  100 experimets, but it's expected to grow fast.
> > All the experimets will be on Affimetrix equipments, so I have no problems
> > in representing the technological platform, since I may provide some
> > template to XML and make the DB compliant even with DB designed to take
> > care of data from different technologies.
> > But I may have troubles in designing the DB schema, becouse I'm not a
> > biologist and here bioogists are not informatics... so I see there may be
> > some comunication problem, possible leading in some incomplete schema.
> > So I decided to start from a good base, such as EBI schema (but I'll think
> > about your one too, which one do you suggest ? ) and customize it.
> Well, we obviously like OUR schema, but if you're a DB person looking at
> schema's to evaluate, you might also look at Stanford's (as a guide, but it
>  is apparently based and quite heavily dependent on Oracle technology):
We are quite oriented to open source technolgy here... so I don't think we'll
use Oracle! 

> > The goal would be to have a standard interface to share data with other DB
> > and with analytical tools too.
> > Which interface do you use to your tools, they are based on your DB-schema,
> > or on GEML ?
> > Is there any infrastructure to share data among DB ?
>  Ours are based on the GeneXML spec (but when MAML gets firmed up enough to
>  use, we'll certainly convert to using that).  We also provide tools written
>  in Perl to support GeneXML exchange among GEneX databases and our Curation
>  Tool (also Open Source)
Any schema the DB will be based on, it will probably feature your XML interface
(whatever! :). 

> You might also want to join the genex-dev mailing list to see how things are
>  moving - and certainly if you want to use GeneX as a basis for your own
>  system.  We'd especially like feedback from people who have Database
>  experience. Sign up via:
thankyou very much for your help,
after the 20, I'm sure I'll subscribe and ask, and ask, and ask....


Andrea Splendiani

[GeneX-dev] harwin up and down for the next while

[Harry M. <man...@ho...>]

I have to reboot harwin to get a new kernel running (linux 2.6 allows for swapping running kernels) to support postgres 7 (damn that Jason!), so please call me right away if you need a cvs checkin/out
in the next few minutes...

At 3:00 Pacific time, I'm going to reboot - should go smoothly, but I reboot so rarely that I may have forgotten how to do it... ;)

-- 
Cheers,
Harry

Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... || man...@ho...

Re: [GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris

[Lonnie M. <lx...@nc...>]

I agree Todd.  This will go away after we implement ssl.  This should be one of our priorities after we implement interim saves.

Lonny
> 
> The browser location might be a good thing to add to curation tool
> properties file.
> 
> Of course, this class of problems would go away if we implemented direct SSL
> connections between curation tool and web server.
> 
> ----- Original Message -----
> From: "Harry Mangalam" <man...@ho...>
> To: <gen...@li...>
> Cc: <qkn...@ad...>
> Sent: Wednesday, February 07, 2001 11:30 AM
> Subject: Re: [GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris
> 
> 
> > I'm pretty sure he means the Curation Tool  (sorry, I should have
> clarified before posting).
> >
> > hjm
> >
> >
> > "Greg D. Colello" wrote:
> > >
> > > Quang:
> > >
> > > I assume you must be running the Curation Tool, because otherwise you
> would have
> > > opened the browser manually to do genex web site access.
> > >
> > > The Curation Tool launches the browser automatically to do either
> download or
> > > upload. This should happen on either the PC or Solaris.
> > >
> > > But it would help to have a more precise definition of what exact
> program you
> > > running. "When running GeneX on Solaris" is not specific enough, since
> GeneX is
> > > just the generic name of our whole suite of tools. Which specific tool
> were you
> > > trying to use?
> > >
> > > Greg
> > >
> > > >From: Harry Mangalam <man...@ho...>
> > > >X-Accept-Language: en
> > > >MIME-Version: 1.0
> > > >To: Quang Nguyen <qkn...@ad...>, genexdev at SF
> > > <gen...@li...>
> > > >Content-Transfer-Encoding: 7bit
> > > >Subject: [GeneX-dev] Bug?: Curation Tool won't start netscape on
> Solaris
> > > >Date: Wed, 07 Feb 2001 10:00:00 -0800
> > > >
> > > >Hi Quang,
> > > >
> > > >   I'm forwarding this to the genex dev list so that the developers are
> made
> > > aware of it.  The problem looks to b e the different application start
> > > mechanisms in Windows vs Solaris, but this should be
> > > >pretty easy to resolve.
> > > >
> > > >Incidentally, what's NACS?
> > > >
> > > >Harry
> > > >
> > > >
> > > >Quang Nguyen wrote:
> > > >>
> > > >> Harry:
> > > >>
> > > >> I don't this is a bug but I think you should know about it.  When
> running
> > > >> genex on solaris, you need to open a web browser before you can start
> > > >> genex because genex does not open it automatically like it does in
> the PC
> > > >> version, or the cause of this problem is that solaris was install
> > > >> through NACS and they have netscape in a different path.  Is there a
> way
> > > >> to fix this problem (i.e., making a symbolic link) so when genex
> starts
> > > >> it opens netscape automatically?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Quang K. Nguyen
> > > >> UCI
> > > >> HHMI Informatics & Computer Core Facility
> > > >> email: qkn...@uc...
> > > >> phone: 949-824-4789
> > > >> fax: 949-824-2688
> > > >
> > > >--
> > > >Cheers,
> > > >Harry
> > > >
> > > >Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... ||
> man...@ho...
> > > >
> > > >_______________________________________________
> > > >Genex-dev mailing list
> > > >Gen...@li...
> > > >http://lists.sourceforge.net/lists/listinfo/genex-dev
> > >
> > > _______________________________________________
> > > Genex-dev mailing list
> > > Gen...@li...
> > > http://lists.sourceforge.net/lists/listinfo/genex-dev
> >
> > --
> > Cheers,
> > Harry
> >
> > Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... ||
> man...@ho...
> >
> > _______________________________________________
> > Genex-dev mailing list
> > Gen...@li...
> > http://lists.sourceforge.net/lists/listinfo/genex-dev
> >
> 
> 
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> http://lists.sourceforge.net/lists/listinfo/genex-dev
> 

[GeneX-dev] Fwd: [XDF-dev] XDF Java Package Beta Released!

[<ja...@op...>]

Hey All,

This isn't directly related, but the XDF package is something that we
should be aware of in any redesign of GeneXML. It now has both a perl
and Java API.

jas.

Re: [GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris

[Todd P. <tf...@nc...>]

The browser location might be a good thing to add to curation tool
properties file.

Of course, this class of problems would go away if we implemented direct SSL
connections between curation tool and web server.

----- Original Message -----
From: "Harry Mangalam" <man...@ho...>
To: <gen...@li...>
Cc: <qkn...@ad...>
Sent: Wednesday, February 07, 2001 11:30 AM
Subject: Re: [GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris


> I'm pretty sure he means the Curation Tool  (sorry, I should have
clarified before posting).
>
> hjm
>
>
> "Greg D. Colello" wrote:
> >
> > Quang:
> >
> > I assume you must be running the Curation Tool, because otherwise you
would have
> > opened the browser manually to do genex web site access.
> >
> > The Curation Tool launches the browser automatically to do either
download or
> > upload. This should happen on either the PC or Solaris.
> >
> > But it would help to have a more precise definition of what exact
program you
> > running. "When running GeneX on Solaris" is not specific enough, since
GeneX is
> > just the generic name of our whole suite of tools. Which specific tool
were you
> > trying to use?
> >
> > Greg
> >
> > >From: Harry Mangalam <man...@ho...>
> > >X-Accept-Language: en
> > >MIME-Version: 1.0
> > >To: Quang Nguyen <qkn...@ad...>, genexdev at SF
> > <gen...@li...>
> > >Content-Transfer-Encoding: 7bit
> > >Subject: [GeneX-dev] Bug?: Curation Tool won't start netscape on
Solaris
> > >Date: Wed, 07 Feb 2001 10:00:00 -0800
> > >
> > >Hi Quang,
> > >
> > >   I'm forwarding this to the genex dev list so that the developers are
made
> > aware of it.  The problem looks to b e the different application start
> > mechanisms in Windows vs Solaris, but this should be
> > >pretty easy to resolve.
> > >
> > >Incidentally, what's NACS?
> > >
> > >Harry
> > >
> > >
> > >Quang Nguyen wrote:
> > >>
> > >> Harry:
> > >>
> > >> I don't this is a bug but I think you should know about it.  When
running
> > >> genex on solaris, you need to open a web browser before you can start
> > >> genex because genex does not open it automatically like it does in
the PC
> > >> version, or the cause of this problem is that solaris was install
> > >> through NACS and they have netscape in a different path.  Is there a
way
> > >> to fix this problem (i.e., making a symbolic link) so when genex
starts
> > >> it opens netscape automatically?
> > >>
> > >> Thanks,
> > >>
> > >> Quang K. Nguyen
> > >> UCI
> > >> HHMI Informatics & Computer Core Facility
> > >> email: qkn...@uc...
> > >> phone: 949-824-4789
> > >> fax: 949-824-2688
> > >
> > >--
> > >Cheers,
> > >Harry
> > >
> > >Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... ||
man...@ho...
> > >
> > >_______________________________________________
> > >Genex-dev mailing list
> > >Gen...@li...
> > >http://lists.sourceforge.net/lists/listinfo/genex-dev
> >
> > _______________________________________________
> > Genex-dev mailing list
> > Gen...@li...
> > http://lists.sourceforge.net/lists/listinfo/genex-dev
>
> --
> Cheers,
> Harry
>
> Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... ||
man...@ho...
>
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> http://lists.sourceforge.net/lists/listinfo/genex-dev
>

Re: [GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris

[Harry M. <man...@ho...>]

I'm pretty sure he means the Curation Tool  (sorry, I should have clarified before posting).

hjm


"Greg D. Colello" wrote:
> 
> Quang:
> 
> I assume you must be running the Curation Tool, because otherwise you would have
> opened the browser manually to do genex web site access.
> 
> The Curation Tool launches the browser automatically to do either download or
> upload. This should happen on either the PC or Solaris.
> 
> But it would help to have a more precise definition of what exact program you
> running. "When running GeneX on Solaris" is not specific enough, since GeneX is
> just the generic name of our whole suite of tools. Which specific tool were you
> trying to use?
> 
> Greg
> 
> >From: Harry Mangalam <man...@ho...>
> >X-Accept-Language: en
> >MIME-Version: 1.0
> >To: Quang Nguyen <qkn...@ad...>, genexdev at SF
> <gen...@li...>
> >Content-Transfer-Encoding: 7bit
> >Subject: [GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris
> >Date: Wed, 07 Feb 2001 10:00:00 -0800
> >
> >Hi Quang,
> >
> >   I'm forwarding this to the genex dev list so that the developers are made
> aware of it.  The problem looks to b e the different application start
> mechanisms in Windows vs Solaris, but this should be
> >pretty easy to resolve.
> >
> >Incidentally, what's NACS?
> >
> >Harry
> >
> >
> >Quang Nguyen wrote:
> >>
> >> Harry:
> >>
> >> I don't this is a bug but I think you should know about it.  When running
> >> genex on solaris, you need to open a web browser before you can start
> >> genex because genex does not open it automatically like it does in the PC
> >> version, or the cause of this problem is that solaris was install
> >> through NACS and they have netscape in a different path.  Is there a way
> >> to fix this problem (i.e., making a symbolic link) so when genex starts
> >> it opens netscape automatically?
> >>
> >> Thanks,
> >>
> >> Quang K. Nguyen
> >> UCI
> >> HHMI Informatics & Computer Core Facility
> >> email: qkn...@uc...
> >> phone: 949-824-4789
> >> fax: 949-824-2688
> >
> >--
> >Cheers,
> >Harry
> >
> >Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... || man...@ho...
> >
> >_______________________________________________
> >Genex-dev mailing list
> >Gen...@li...
> >http://lists.sourceforge.net/lists/listinfo/genex-dev
> 
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> http://lists.sourceforge.net/lists/listinfo/genex-dev

-- 
Cheers,
Harry

Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... || man...@ho...

Re: [GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris

[Greg D. C. <gd...@nc...>]

Quang:

I assume you must be running the Curation Tool, because otherwise you would have 
opened the browser manually to do genex web site access.

The Curation Tool launches the browser automatically to do either download or 
upload. This should happen on either the PC or Solaris.

But it would help to have a more precise definition of what exact program you 
running. "When running GeneX on Solaris" is not specific enough, since GeneX is 
just the generic name of our whole suite of tools. Which specific tool were you 
trying to use?

Greg

>From: Harry Mangalam <man...@ho...>
>X-Accept-Language: en
>MIME-Version: 1.0
>To: Quang Nguyen <qkn...@ad...>, genexdev at SF 
<gen...@li...>
>Content-Transfer-Encoding: 7bit
>Subject: [GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris
>Date: Wed, 07 Feb 2001 10:00:00 -0800
>
>Hi Quang,
>
>   I'm forwarding this to the genex dev list so that the developers are made 
aware of it.  The problem looks to b e the different application start 
mechanisms in Windows vs Solaris, but this should be
>pretty easy to resolve.
>
>Incidentally, what's NACS?
>
>Harry
>
>
>Quang Nguyen wrote:
>> 
>> Harry:
>> 
>> I don't this is a bug but I think you should know about it.  When running
>> genex on solaris, you need to open a web browser before you can start
>> genex because genex does not open it automatically like it does in the PC
>> version, or the cause of this problem is that solaris was install
>> through NACS and they have netscape in a different path.  Is there a way
>> to fix this problem (i.e., making a symbolic link) so when genex starts
>> it opens netscape automatically?
>> 
>> Thanks,
>> 
>> Quang K. Nguyen
>> UCI
>> HHMI Informatics & Computer Core Facility
>> email: qkn...@uc...
>> phone: 949-824-4789
>> fax: 949-824-2688
>
>-- 
>Cheers,
>Harry
>
>Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... || man...@ho...
>
>_______________________________________________
>Genex-dev mailing list
>Gen...@li...
>http://lists.sourceforge.net/lists/listinfo/genex-dev

[GeneX-dev] Bug?: Curation Tool won't start netscape on Solaris

[Harry M. <man...@ho...>]

Hi Quang,

   I'm forwarding this to the genex dev list so that the developers are made aware of it.  The problem looks to b e the different application start mechanisms in Windows vs Solaris, but this should be
pretty easy to resolve.

Incidentally, what's NACS?

Harry


Quang Nguyen wrote:
> 
> Harry:
> 
> I don't this is a bug but I think you should know about it.  When running
> genex on solaris, you need to open a web browser before you can start
> genex because genex does not open it automatically like it does in the PC
> version, or the cause of this problem is that solaris was install
> through NACS and they have netscape in a different path.  Is there a way
> to fix this problem (i.e., making a symbolic link) so when genex starts
> it opens netscape automatically?
> 
> Thanks,
> 
> Quang K. Nguyen
> UCI
> HHMI Informatics & Computer Core Facility
> email: qkn...@uc...
> phone: 949-824-4789
> fax: 949-824-2688

-- 
Cheers,
Harry

Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... || man...@ho...

Re: [GeneX-dev] DB Update script ...

[<ja...@op...>]

Hi Elena,

Harry and I ran into an issue that needed to be fixed. It turns out
that in previous DB releases all tables were owned by the Postgres DB
user 'postgres', and not the user 'genex'. I needed to write a script
that would change this. That is now done.

I am uploading v0.9 of the DB, and will test running an installation
from scratch. If it succeeds I will cut a release of the Server
tonight. 

jas.

eb...@uc... writes:

> Is the update available yet as I need help just as badly as Michael 

Re: [GeneX-dev] DB Update script ...

[<eb...@uc...>]

Hi Jas

Is the update available yet as I need help just as badly as Michael 

Thanks 

Elena


---------Included Message----------
>Date: 05 Feb 2001 23:08:23 -0700
>From: "Jason E. Stewart" <ja...@op...>
>Reply-To: <gen...@li...>
>To: <gen...@li...>
>Subject: [GeneX-dev] DB Update script ...
>
>... is finished.
>
>It is now possible to update a 0.8.1 DB to a version 0.9 DB. This is
>not 100% of what I wanted to do, it isn't even 70%. But at least it
>will help people like Michael out.
>
>I still need a way of converting a public-dump into an NCGR specific
>dump by adding Contact and UserSec entries.
>
>I will cut DB-0.9, and Server-1.0.0-pre8 tomorrow, install and test
>them on genesoup. And if they fly, I'll upload them to SourceForge
>(I'll even add it to the WWW page ;-) and post them to the list.
>
>jas.
>
>_______________________________________________
>Genex-dev mailing list
>Gen...@li...
>http://lists.sourceforge.net/lists/listinfo/genex-dev
>
---------End of Included Message----------

Re: [GeneX-dev] SourceForge web site good to go

[<ja...@op...>]

Hi Leo,

Thanks for finding that. I fixed the link to work.

Cheers,
jas.

"Heijden, L.A.M. van der" <L.A...@pl...> writes:

> I found a dead link on http://genex.sourceforge.net:
> 
> http://genex.sourceforge.net/GeneX.Perl.modules.html
> 

Re: [GeneX-dev] SourceForge web site good to go

[Harry M. <man...@ho...>]

Hi Leo (and welcome to the list),

Hah!  And they say Open Source doesn't work... ;)

Thanks for the eagle eyeball. It's fixed:

http://genex.sourceforge.net/GeneX.Perl.modules
(not an HTML..yet)

hjm


"Heijden, L.A.M. van der" wrote:
> 
> Hi,
> 
> I found a dead link on http://genex.sourceforge.net:
> 
> http://genex.sourceforge.net/GeneX.Perl.modules.html
> 
> Leo van der Heijden,
> Plant Research International,
> Wageningen, The Netherlands
> 
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> http://lists.sourceforge.net/lists/listinfo/genex-dev

-- 
Cheers,
Harry

Harry J Mangalam -- (949) 856 2847 (v&f) -- hj...@nc... || man...@ho...

RE: [GeneX-dev] SourceForge web site good to go

[Heijden, L.A.M. v. d. <L.A...@pl...>]

Hi,

I found a dead link on http://genex.sourceforge.net:

http://genex.sourceforge.net/GeneX.Perl.modules.html

Leo van der Heijden,
Plant Research International,
Wageningen, The Netherlands