genex-dev — GeneX developers

[GeneX-dev] Re: User Data insertion

[<ja...@op...>]

jw...@gm... writes:

> More Executive Decisions: Users have to be allowed to insert data - we
> could limit upload size per user (space above default to be negotiated
> with the Administrator or Curator). I would prefer a task-prioritized
> scheduler to a timed scheduler, it seems more efficient. Is it
> possible to have a graphical `current usage' widget for users so that
> you can see if a fairly extensive task is in process (so come back
> later)? Once we figure it out for ourselves it would also be useful to
> provide an estimate of the time some of the more intensive tasks will
> likely take (eg loading 20 U133a chips will take 2 hours - so come
> back later).

Ok, I like the scheduler idea. The graphical widget is definately
possible, but as always it's time and priorities that must be
decided. 

The size estimates are a good idea, but that depends a lot on the load
on the machine (how many users are hitting the DB), and what type of
machine it is. We can certainly give examples of what we've found that
can be used as guidelines.

> Future Executive Decision: I would like to wait to split db and
> analysis functions until we have a demonstrated need, and also
> appropriate resources. We tried this development path at NCGR and got
> bogged down in architecture decisions before we ever had data in the
> first place (of course, there was that Curation Tool.....)

Agreed.

Cheers,
jas.

[GeneX-dev] Re: Modifying Archived data

[<ja...@op...>]

jw...@gm... writes:

> 1. Executive Decisions: I completely agree that neither PBA nor MBA
> should be modifiable except by the data owner, through the curator -

Actually, they should be able to modify the meta-data in the MBA and
PBA, but not the 'data' in the MBA data matrix. They need to modify
and update their annotations (which are stored as part of the MBA and
PBA), but we don't want users accidentally modifying or deleting
data. 

> and we have to decide about cascading effects when this does happen-
> if one of these is modified what do we do about downstream DBAs? 

This is why I decided to simplify our life and make it archive only. 

If the user wants to change it, it should be copied and the copy
modified. 

> It is likely that the results can no longer be verified by applying
> the specified protocol to the substituted data matrix. It is also
> likely that if we delete them someone will get really pissed off. It
> is also likely that additional users will not want to build more
> analyses on top of them, at least not without elaborate warnings.

correct.

Cheers,
jas.

Re: [GeneX-dev] Jasons Executive Decisions, Data Workflow comments, Miscellaneous Questions

[<ja...@op...>]

jw...@gm... writes:

> Question - where are we storing the PBA images?  

Good question.

Bad answer, currently, we are not storing them at all.

As soon as we decided to do this, we can choose a directory in which
data is archived in the file system under the genex installation
directory (e.g. /usr/local/genex/archives) and store them there when
the user annotates the PBA. We can create a very simple 'Add Images to
PBA' application for them.

Because a PBA can have multiple images (depending on the number of
channels it is being scanned at), we will need a seperate Image
table. 

  => Schema change.

I've added this to the TODO list.

> If someone wants to load MBAs for which we don't have images, how is
> the original referenced? A URL, a citation? 

Good question.

There is currently only one column in the DB to store this type of
information, and it is in the Experiment table, and it has the
unfortunate name 'local_accession' - which is simply a text string. We
should probably give it a more useful name like 'external_reference'
or something, and we may want to make it a foreign key to the
DatabaseEntry table. That way, they get a lot of ability to document
were it came from - an official DB that uses accession numbers or
merely a URL.

The Experiment table also has a Contact table link that is used to
indicate an external data provider's contact information, and it has
the Citation table link that could be used for any publication.

However, what about other pieces of data like BioAssay's or
Biomaterial's or Protocol's - each of these can be obtained from
external sources, and users may want to reference these in some
fashion. I think they should also be given an optional DatabaseEntry
fkey to track where they came from.

> Is it optional or required to reference this?

My mantra is 'everything should be optional unless it breaks genex' -
this doesn't break anything, so to me, it's optional.

Cheers,
jas.

Re: [GeneX-dev] MBA and PBA images

[<ja...@op...>]

jw...@gm... writes:

> Should the Measured BioAssay be required to reference an image? 

No, because many times users will download data from other people and
they don't want to bother with images - they just want to include
other researchers data for their own analyses.

In general, genex should not require something unless something within
genex will break without it. 

My experiences says that if you restrict things to much, you limit how
creative people can use your system in ways you didn't imagine.

> Your explanation says it OFTEN uses images, referenced through PBA,

Correct, images are PBA Data (according to MAGE). The images are used
to create the MBA Data which we load into genex.

> but this seems fuzzy. 

Not every experiment in the DB will have images.

> How are we going to discriminate the Derived Bioassay from the
> Measured BioAssay unless we impose this constraint?  

We load MBA Data using the MBA Data loader. That creates an MBA in the
DB. 

If we want to, we could also create a DBA Data loader that loads data
directly into the scratch table and creates a DBA, but that seems
lower priority right now.

> Or alternatively, when would we want to call something an MBA that
> did not reference an image?

Because someone loaded data from an external source and didn't want
the images.

Cheers,
jas.

Re: [GeneX-dev] Problems using views in OOo, even as 'genex'

[<ja...@op...>]

Harry Mangalam <hj...@ta...> writes:

> Jeez, I dunno how you found that - I looked for that page for about an
> hour today.^$%$#%%^

google for 'odbc postgres views' 

(Google Hacks by O'Reilly is your friend ;-)

> Anyway, yes, it works with the .ctid and .oid colums tagged on.
> They're empty values as far as I can see, but PG needs them, at least
> for ODBC.  

Seems to have data in my DB:

genex2=# select oid,ctid from genex_role;
  oid   | ctid  
--------+-------
 127383 | (0,1)
 126795 | (0,2)
 126898 | (0,3)
 126987 | (0,4)
 127055 | (0,5)
 127109 | (0,6)
 127450 | (0,7)
(7 rows)

> I used both, so I don't know if the ctid is required for the linux
> version and not the Windows version or not, but from the error
> messages, it looks like both will be needed to service requests from
> Lin & Win, so best to just add them both.
>
> Good!! almost there!!
>
> Finshed with the Linux version of the ODBC doc, will send it off in a minute.
>
> Have to verify how things work on OOo and Excel in Windows now - ugh.  but nec.
> And this approach really looks VERY promising.
>
> Multiple, Fabulous front ends to GeneX!!

Definately very cool! I think this is really a tremendous piece of
work that will benifit many projects.

Cheers,
jas.

Re: [GeneX-dev] Problems using views in OOo, even as 'genex'

[Harry M. <hj...@ta...>]

Jeez, I dunno how you found that - I looked for that page for about an hour 
today.^$%$#%%^

Anyway, yes, it works with the .ctid and .oid colums tagged on.  They're empty 
values as far as I can see, but PG needs them, at least for ODBC.  I used both, 
so I don't know if the ctid is required for the linux version and not the 
Windows version or not, but from the error messages, it looks like both will be 
needed to service requests from Lin & Win, so best to just add them both.

Good!! almost there!!

Finshed with the Linux version of the ODBC doc, will send it off in a minute.

Have to verify how things work on OOo and Excel in Windows now - ugh.  but nec.
And this approach really looks VERY promising.

Multiple, Fabulous front ends to GeneX!!

hjm


Jason E. Stewart wrote:
> Harry J Mangalam <hj...@ta...> writes:
> 
> 
>>I've further debugged the problem in getting the views to work.  we
>>may have to add another column to carry alon in the views -that of
>>the ctid.
> 
> 
> Hey Harry,
> 
> Apparently it's ctid and oid:
> 
>   http://archives.postgresql.org/pgsql-odbc/2003-06/msg00019.php
> 
> Simply including those rows in the view seems to make it work. Go
> ahead and make a test view and see if it still works, and I'll include
> those columns for scratch table views. 
> 
> Modifying views doesn't affect the underlying data, so it won't affect
> a schema freeze - updating view definitions will only affect the
> analysis tools
> 
> Cheers,
> jas.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by Sleepycat Software
> Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to deliver
> higher performing products faster, at low TCO.
> http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev
> 

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

Re: [GeneX-dev] Problems using views in OOo, even as 'genex'

[<ja...@op...>]

Harry J Mangalam <hj...@ta...> writes:

> I've further debugged the problem in getting the views to work.  we
> may have to add another column to carry alon in the views -that of
> the ctid.

Hey Harry,

Apparently it's ctid and oid:

  http://archives.postgresql.org/pgsql-odbc/2003-06/msg00019.php

Simply including those rows in the view seems to make it work. Go
ahead and make a test view and see if it still works, and I'll include
those columns for scratch table views. 

Modifying views doesn't affect the underlying data, so it won't affect
a schema freeze - updating view definitions will only affect the
analysis tools

Cheers,
jas.

[GeneX-dev] Problems using views in OOo, even as 'genex'

[Harry J M. <hj...@ta...>]

Hi Jason,
I've further debugged the problem in getting the views to work.  we may hav=
e=20
to add another column to carry alon in the views -that of the ctid.

The problem is that when logged in as genex (to matrix as it happens, but i=
t's=20
generic with all genex installations [by the way could you send me the gene=
x=20
password for the DB on genex2?])  I can't get the views - I can only get th=
e=20
tables.  If I try to see the views, I get the OOo error:

The data content could not be updated
[unixODBC]ERROR: column "ctid" does not exist.

=46rom the postgres release notes, each table has a few system columns,=20
including ctid:
<quote>
ctid

 The physical location of the tuple within its table. Note that although th=
e=20
ctid can be used to locate the tuple very quickly, a row's ctid will change=
=20
each time it is updated or moved by VACUUM FULL. Therefore ctid is useless =
as=20
a long-term row identifier. The OID, or even better a user-defined serial=20
number, should be used to identify logical rows.=20
</quote>
If the ctid column is not being copied to or isn't accessible in the view,=
=20
OOo/ODBC can't retrieve the data.  I'm not sure if this is a bug or just th=
e=20
way it's supposed to be.  I'll keep looking into this.

Also, while the readonly user can pull the names of the tables and views fr=
om=20
a genex DB via OOo/ODBC, he can't pull any of the actual data from it.  I g=
et=20
a permissions problems.  I haven't identified this any more deeply just yet=
,=20
this is just a head's up.

On the other hand the OOo methods of building queries and being able to use=
=20
them is just .. GREAT!  After a bit, it becomes very easy to build the=20
queries, and then they can be saved as icons to be executed by a single=20
click.  The query icons can be dragged to an open spreadsheet where they ar=
e=20
executed and the SS gets populated by the answer.  It is really remarkable.=
=20
We do need a way to be able to export/import the data source configuration=
=20
back and forth tho and=20

=2D-=20
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...=20
            <<plain text preferred>>

[GeneX-dev] Schema changes not finished

[<ja...@op...>]

Hey All,

I was hoping to have the final schema changes finished and tested by
today, but it took quite some time to reactivate the regression tests,
upon which I discovered a nasty schema bug, so I'm running behind
schedule.

I have made the BioAssayLink addition to the schema and I'm in the
middle of testing it - using PBA -> MBA -> DBA, and I'm currently
stuck at the DBA step, getting wierd DB errors.

I am not going to guess what a timeline is, so I will keep you posted.

Cheers,
jas.

[GeneX-dev] Regression test active again

[<ja...@op...>]

Hey All,

Phew! It was a lot more work than I had hoped, but it's now
complete. Running 'make test' from Perl/Bio-Genex now produces the
following (final) output:

   All tests successful, 2 subtests skipped.
   Files=63, Tests=3021, 66 wallclock secs (56.14 cusr +  3.93 csys = 60.07 CPU)

In the process of re-enabling the tests I discovered a serious++ bug
in the naming of foreign keys in the Perl API - just another reason
why I believe (as does Harry) that even if you can't do XP, you should
still write the tests before you write the code.

I can't yet commit the new code, because it's got half-finished schema
changes...

Cheers,
jas.

[GeneX-dev] Bioinformatics Open Source Conference - ABSTRACT DEADLINE 5/5/2004

[Brandon K. <ki...@ca...>]

Hi Jason,
    You are probably already away of this, but I thought I should 
mention it just in case. http://open-bio.org/bosc2004/

-Brandon

Re: [GeneX-dev] GeneSpring docs

[Harry M. <hj...@ta...>]

saw it, will do.


Jason E. Stewart wrote:
> Hey Harry,
> 
> It would be nice to use GeneSpring to test out the new Scratch table
> float => integer stuff.
> 
> I made a task for updating the old docs for this, and adding it to the
> docs page.

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

[GeneX-dev] [ genex-Bugs-947522 ] Ensure that Bio-Genex 'make test' is functional

[SourceForge.net <no...@so...>]

Bugs item #947522, was opened at 2004-05-04 01:54
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=947522&group_id=16453

Category: Bio::Genex Perl API
Group: Genex-2
Status: Open
Resolution: None
Priority: 9
Submitted By: Jason E. Stewart (jason_e_stewart)
Assigned to: Jason E. Stewart (jason_e_stewart)
Summary: Ensure that Bio-Genex 'make test' is functional

Initial Comment:
We have not been running the automated regression tests
on the DB in a long time. To ensure that changes are
not breaking existing code we need to resume testing.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=947522&group_id=16453

[GeneX-dev] Documentation Tasks

[<ja...@op...>]

Hey All,

In preparation for the Beta Release, we need to ensure that the
documentation is up-to-date. I took Jennifer's email from Feb 24 and I
made a number of tasks on the SF Documentation tracker so that we can
keep track of what pieces need adding/updating.

I made Jennifer overall coordinator for the task - so she has full
power to bother us if we're not getting the docs done in time. I made
a very generous time frame for most of these docs for the end of next
week.

Jennifer, if you need some docs sooner, let us know.

Cheers,
jas.

[GeneX-dev] GeneSpring docs

[<ja...@op...>]

Hey Harry,

It would be nice to use GeneSpring to test out the new Scratch table
float => integer stuff.

I made a task for updating the old docs for this, and adding it to the
docs page.

Cheers,
jas.

[GeneX-dev] Scratch table description

[<ja...@op...>]

Hey Harry,

You and I had a GAIM session a while back in which I gave a lengthy
discussion of:
1) how to hook up external processing scripts
2) the scratch table and scratch views

Do you still have the log for that? It would be good to convert it
into a document under the document tree.

I've created a SF Task for this.

Cheers,
jas.

Re: [GeneX-dev] Data insert into GeneX?

[Harry M. <hj...@ta...>]

Hi Muhammed,

My apologies for the delay in responding - you have to be a member to post to 
the group so your post was held up until I was warned of it being held and then 
I forgot to answer it right away - bad, bad.

Regarding your use of GeneX, it looks like you're using an early GeneX-1 schema 
- the latest one is considerably more advanced and flexible (at the cost of 
being more complex).

If your data is as simple as you describe, and you only want to store it in a 
database, GeneX-2 will be overkill for your efforts.  You might consider using 
either a much simpler schema with perhaps only a simgle table or if you want to 
make use of some incoporated analyses, you might try the BASE gene expression 
DB, which I understand is quite simple to set up (tho is not scalable as GeneX) 
and comes with some analyses already set up:

http://base.thep.lu.se

If you have money to spare, you might also consider the J-Express Db which is 
sold by molmine:

http://www.molmine.com

or iobion's genetraffic.


If you think your data will expand and will be required to be MIAME/MAGE 
compatible, then please sign up for the development listservs and post more 
info, and we'll try to answer your questions in a more timely manner.

Harry


Muhammad wrote:
> Hallo,
> I want to insert Gene experiment data into GeneX Database.
> for this purpose, i have choosen a small schema.
> As attechment you will find choosed schema, where i am intended
> to insert data.
>  
> 
> We have gene experiment data in two flate files. The first file contains 
> clinical information and the 
> scecond spots information. 
> In first file, we have four following clinical properties.
> 1.	sample name
> 2.	Translocation
> 3.	Treatment type
> 4.	Outcome information
> 
> like following formate:
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Sample name	translocateion	type	outcome
> BCR-ABL-#1	BCR-ABL		T13A	Censored
> BCR-ABL-#2	BCR-ABL		T13B	Censored
> BCR-ABL-#3	BCR-ABL		T13B	Censored
> BCR-ABL-#4	BCR-ABL		T11	NA
> BCR-ABL-#5	BCR-ABL		T12	NA
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> And in the other flate file, we have following spots information.
> 4.	spot_name
> 5.	Normalized intesity (expression) of on a spot with its belonging 
> Sample_name.
> 
> 
> like following formate:
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Gene_name	BCR.ABL..1	BCR.ABL..2	BCR.ABL..3	BCR.ABL..4	BCR.ABL..5
> 1000_at		1.44316890296237	1.34035075057766	1.48303546191075	1.46672518104584	1.
> 48830338653164	
> 1001_at		0.978063915142567	1.08129954743597	1.09047133840959	0.966733966954692	0.
> 92340706624928	
> 1002_f_at	0.888879088423292	0.891881799999218	0.79746788484822	0.
> 95309567091632	0.785177495990561
> 1003_s_at	1.22144402768475	1.27740380645519	1.14955589812956	1.28440406219124	1.
> 17954677735482
> 1004_at		1.19813226946593	1.11859689741790	1.11275118624824	1.16928072863073	1.
> 11182581513464
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> I am intended to insert data as follow:
> -"Sample name" and "Translocation" will be insert into Title Attribute 
> of table
> Protocol.
> -In type Attribute will be insert treatment type.
> -In Text Attribute will insert the values of "Sample_id" and "translocation".
> 
> -Gene_id will insert into "spot identifier" Attribute of table AL_Spots.
> -Intensity values will insert into "spot_value" Attribute of table 
> AM_Spot. 
> -ArrayLayout table will contain the name of Array in "name" Attribute.
> -"Comment_prot" of Sample table will contain the "outcome" values 
> e.g Censored.
> 
> i want to know, is the choosed DB-Schema suitable for inserting the 
> above data?
> or i should consider also other tables or other links table?
> 
> I will be thank full for your kindly reply and worthfull comments.
> 
> With best regards,
> Muhammad Ghiyas
> 
> 
> ------------------------------------------------------------------------
> 
> 
> ------------------------------------------------------------------------
> 
> 
> 
> ===================================================================
> EASY and FREE access to your email anywhere: http://Mailreader.com/
> ===================================================================

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

[GeneX-dev] Re: Matrix update

[Harry M. <hj...@ta...>]

OK - then let's follow Jason's rec to let him have the 2 days to tidy up the 
schema and Mason stuff - meanwhile we write and tidy the docs for GeneX and the 
Web site.  On Wednesday, I'll refresh matrix and we can start loading data in anger.

hjm



jw...@gm... wrote:
> Hi Harry,
>    I would like matrix to be refreshed; I agree that the changes Jason suggests below add some clarity to the installation process so should be incorporated. I would especially like to be sure that the interface bugs Jason has been batting away at have been fixed on matrix- I ran into a bunch of them on Saturday and I don't want to re-document what has already been fixed.
>    Cheers,
>    Jennifer
> 
> 
> 
> ----- Original Message -----
> From: Harry Mangalam <hj...@ta...>
> Date: Sunday, May 2, 2004 1:18 pm
> Subject: Re: [GeneX-dev] make configure
> 
> 
>>This sounds good - I was going to suggest that we 
>>consolidate/clean up the 
>>recon/configure divergence anyway.
>>
>>And making things clearly genex-related is a good idea for all 
>>kinds of reasons.
>>
>>Jennifer, what's the status of matrix re: data loading?  DO you 
>>want me to 
>>refresh it to get these things incorporated or should we leave it 
>>alone - these 
>>changes are administrative and won't change any 
>>db/schema/interface details.
>>
>>hjm
>>
>>
>>Jason E. Stewart wrote:
>>
>>>Hey all,
>>>
>>>It looks like a change I made to the 'Configure' script on 3/23/2004
>>>got lost when recon.pl was split off - I had renamed the 'readonly'
>>>user to 'sessions' because that is the name of the only table that
>>>user has access to. This wasn't obvious because I never use the
>>>hard-coded name of any user in the API - I always use the Perl
>>>Config.pm variable, e.g. $Bio::Genex::Config->{GENEX_TEST_USER}.
>>>
>>>But it showed up when I was helping Durga debug her installer 
>>
>>problem.> 
>>
>>>I'm proposing the following changes:
>>>1) Rename recon.pl to Configure - make it the 'official' 
>>
>>configuration>    tool. It's had enough testing now, I think it's 
>>stable.> 
>>
>>>2) Make the genex administrative user names clearly genex
>>>   specific. This is in keeping with the role names, e.g. 
>>
>>genex_admin,>    genex_user, etc. So instead of 'sessions' make it 
>>'genex_sessions',>    etc. This way our administrative users don't 
>>conflict with other
>>
>>>   potential users.
>>>
>>>3) Instead of only a test user that has only USER level priveleges,
>>>   add a second user that has CURATOR priveleges as well. This 
>>
>>came up
>>
>>>   when I was adding the sample data set - I needed a user with
>>>   curator priveleges, so I had to give them to the 'test' user.
>>>
>>>These changes are relatively minor, but I believe they will make 
>>
>>long> term maintenance simpler.
>>
>>>I'm asking this to the list, because it will require a 'make
>>>uninstall' for every installation (the old users will need to be
>>>deleted from the DB).
>>>
>>>Cheers,
>>>jas.
>>>
>>>
>>>-------------------------------------------------------
>>>This SF.Net email is sponsored by: Oracle 10g
>>>Get certified on the hottest thing ever to hit the market... 
>>
>>Oracle 10g. 
>>
>>>Take an Oracle 10g class now, and we'll give you the exam FREE.
>>>http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
>>>_______________________________________________
>>>Genex-dev mailing list
>>>Gen...@li...
>>>https://lists.sourceforge.net/lists/listinfo/genex-dev
>>>
>>
>>-- 
>>Cheers, Harry
>>Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
>>            <<plain text preferred>>
>>
>>

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

[GeneX-dev] Matrix update

[<jw...@gm...>]

Hi Harry,
   I would like matrix to be refreshed; I agree that the changes Jason suggests below add some clarity to the installation process so should be incorporated. I would especially like to be sure that the interface bugs Jason has been batting away at have been fixed on matrix- I ran into a bunch of them on Saturday and I don't want to re-document what has already been fixed.
   Cheers,
   Jennifer



----- Original Message -----
From: Harry Mangalam <hj...@ta...>
Date: Sunday, May 2, 2004 1:18 pm
Subject: Re: [GeneX-dev] make configure

> This sounds good - I was going to suggest that we 
> consolidate/clean up the 
> recon/configure divergence anyway.
> 
> And making things clearly genex-related is a good idea for all 
> kinds of reasons.
> 
> Jennifer, what's the status of matrix re: data loading?  DO you 
> want me to 
> refresh it to get these things incorporated or should we leave it 
> alone - these 
> changes are administrative and won't change any 
> db/schema/interface details.
> 
> hjm
> 
> 
> Jason E. Stewart wrote:
> > Hey all,
> > 
> > It looks like a change I made to the 'Configure' script on 3/23/2004
> > got lost when recon.pl was split off - I had renamed the 'readonly'
> > user to 'sessions' because that is the name of the only table that
> > user has access to. This wasn't obvious because I never use the
> > hard-coded name of any user in the API - I always use the Perl
> > Config.pm variable, e.g. $Bio::Genex::Config->{GENEX_TEST_USER}.
> > 
> > But it showed up when I was helping Durga debug her installer 
> problem.> 
> > I'm proposing the following changes:
> > 1) Rename recon.pl to Configure - make it the 'official' 
> configuration>    tool. It's had enough testing now, I think it's 
> stable.> 
> > 2) Make the genex administrative user names clearly genex
> >    specific. This is in keeping with the role names, e.g. 
> genex_admin,>    genex_user, etc. So instead of 'sessions' make it 
> 'genex_sessions',>    etc. This way our administrative users don't 
> conflict with other
> >    potential users.
> > 
> > 3) Instead of only a test user that has only USER level priveleges,
> >    add a second user that has CURATOR priveleges as well. This 
> came up
> >    when I was adding the sample data set - I needed a user with
> >    curator priveleges, so I had to give them to the 'test' user.
> > 
> > These changes are relatively minor, but I believe they will make 
> long> term maintenance simpler.
> > 
> > I'm asking this to the list, because it will require a 'make
> > uninstall' for every installation (the old users will need to be
> > deleted from the DB).
> > 
> > Cheers,
> > jas.
> > 
> > 
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: Oracle 10g
> > Get certified on the hottest thing ever to hit the market... 
> Oracle 10g. 
> > Take an Oracle 10g class now, and we'll give you the exam FREE.
> > http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> > _______________________________________________
> > Genex-dev mailing list
> > Gen...@li...
> > https://lists.sourceforge.net/lists/listinfo/genex-dev
> > 
> 
> -- 
> Cheers, Harry
> Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
>             <<plain text preferred>>
> 
> 

Re: [GeneX-dev] BioAssay's

[<ja...@op...>]

Harry Mangalam <hj...@ta...> writes:

> Jason - that was a great beginner's intro to MAGE and GeneX
> terminology.  I'll try to work it up into a doc to put on the web
> site.   Probably by tomorrow.

Excellent, thanks.

> Aren't docs are a great way to verify whether code reflects reality
> and vice versa. :)

I've discovered this to be the case...

> I've almost completely changed my mind about the relative importance
> of code, docs, and regression testing as I've grown older.
>
> Detailed planning docs 1st, code and regression tests as you go along,
> and once they're stable, document the function.  

Yes, I agree. My feeling something like this:

1) Planning docs - not too detailed, but rich enough to give a
   framework to the entire project.

Followed by cycles of:
2) regression test building
3) code generation
4) upgrading of documentation (if needed).

> Once you have a skeleton set of tests, it becomes easier and easier
> to add to them and once they're set up, you can test everything and
> all interactions at once. (Found out a lot about differences in
> versions of gcc this way).  If you can't do XP, you absolutely need
> automated regressions tests.  If you can do XP, you still need it.

Couldn't agree more...

Believe it or not, genex has had a huge number of regression tests
since the genex-1 days - and I used to run them all the time, but when
I started making huge changes to the schema I got out of the habit of
running them because most of the old test data couldn't be loaded
anymore.

I'll check to see which tests can be reactivated, and then we can make
it a policy to run the regression tests *before* doing an svn commit.

Cheers,
jas.

[GeneX-dev] Upgrade script requirements

[<ja...@op...>]

Hey all,

If Genex had full MAGE-ML export capabilities, our upgrade issues
would be simpler, not solved but simpler. But we don't yet have full
MAGE export - in fact we have next to zero mage export, and only
slightly more MAGE import.

We all agree that we cannot continue the ridiculous policy of
inflicting total data loss on users of genex whenever they upgrade
their DB. So we agree we need a non-destructive we to upgrade genex.

There are currently three pieces to genex:
* DB schema and the DB tables and views created from the schema
* Perl API
* associated applications, Perl scripts, R scripts, Mason pages

Upgrading the final two pieces is already totally non-destructive -
any user can upgrad the API or the associated tools without affecting
any data - this has always been possible for genex-2.

So the only piece that is potentially destructive is modifications to
the schema. In the following [nd] means non-destructive, while [D]
means potentially destructive.

There are a number of possible modifications to the schema:
1) adding new tables [nd]
2) deleting old tables [D]
3) modifying existing tables [D]

At this point, it is not so likely that we will need to remove tables
- I have cleaned out all tables that I think are not needed. Adding
new tables shouldn't affect existing data. So the only major issues
arise when modifying tables. 

The possible modifications are:
1) renaming a table [nd] - postgres supports this in the 'ALTER TABLE
   ...' command so no worries
2) adding a column [nd] - postgres supports this in the 'ALTER TABLE
   ...' command so no worries
3) renaming a column [nd] - postgres supports this in the 'ALTER TABLE
   ...' command so no worries
4) drop a column [D] - goodbye column, goodbye data. 
5) move a column [D] - this is both adding a column and removing a
   column, so it's destructive
5) removing a NOT NULL constraint to a column [nd] - postgres supports
   this in the 'ALTER TABLE ...', so no worries
6) adding a NOT NULL constraint to a column [nd] - postgres supports
   this in the 'ALTER TABLE ...', if the user has already entered
   data and some rows are NULL postgres will give an error, so we have
   to check first and set NULL rows to some reasonable default value
7) changing the data type of a column [D] - An example would be
   making a varchar(12) column to be a varchar(128) column. This is
   adding a new column, copying over the data, removing the old
   column, and renaming the new column. Assuming the old data type can
   be coerced into the new data type - no problem. But if we modify a
   column to be more restrictive, e.g. changing a float column to an
   int, then we could lose data.

So we only have three possible worries dropping a column (OK, so
let's agree we won't drop columns), moving a column, e.g. moving the
mba_fk column from the PhysicalBioAssay table to the BioAssayLink
table, or changing a column to be a more restrictive data type, e.g. a
varchar(128) to a varchar(6) will truncate any long strings already in
the table.

The most likely change is moving a column. 

So all in all, I'm more relieved about this then I was a while back. 

Cheers,
jas.

Re: [GeneX-dev] make configure

[<ja...@op...>]

Harry Mangalam <hj...@ta...> writes:

> Jennifer, what's the status of matrix re: data loading?  DO you want
> me to refresh it to get these things incorporated or should we leave
> it alone - these changes are administrative and won't change any
> db/schema/interface details.

Matrix has a horribly old version of the genex-2 schema. Does it have
any data yet besides the small bit of test data that is autoloaded at
installation time? If there is data loaded let me know, and we can
start the upgrade script sooner.

We've still done what I would call *minimal* testing of genex, and
with that minimal testing we've found some pretty serious deficiencies
in the schema, i.e. the lack of the BioAssayLink table. We haven't
really identified what tests we ought to do to feel confident about
making the beta release.

Locking matrix before we have even a remote belief that the schema is
stable is pointless. Right now, I know the schema is not stable due to
the massive problem I discovered write the BioAssay document, e.g. no
way to discover how BioAssay's were created.

Here's my suggestion. 

1) Final schema cleanup
Let me spend two days cleaning up all remaining schema issues:
* the lost BioAssayLink table
* switch ControlledVocab to OntologyEntry
* user name switches

2) Schema freeze
Do a full DB upgrade on matrix at this point. 

Then we make a schema freeze - we do lots more testing, and log any
bugs, but only priority-9 schema bugs will be accepted for change
pre-beta. Any lower priority schema bugs we find will only be accepted
post-beta. 

3) Upgrade script (if necessary)
If there are any schema changes that are added post-freeze I will have
to create an upgrade script that saves any affected data in the DB,
does the upgrade, and then reloads the data. 

At this point we start tracking DB version numbers, so that way we can
have upgrade scripts that are version specific. So that any change to
the schema XML files causes us to bump the DB version (I use ISO-8660
date version numbers for all my files nowadays,e.g. '20040503.0'). The
decimal point is for multiple changes an a single day.

We are going to need to do this in any case, so the sooner we start
the better off we'll be.

Cheers,
jas.

Re: [GeneX-dev] BioAssay's

[Harry M. <hj...@ta...>]

Jason - that was a great beginner's intro to MAGE and GeneX terminology.  I'll 
try to work it up into a doc to put on the web site.   Probably by tomorrow.

Aren't docs are a great way to verify whether code reflects reality and vice 
versa. :)

I've almost completely changed my mind about the relative importance of code, 
docs, and regression testing as I've grown older.

Detailed planning docs 1st, code and regression tests as you go along, and once 
they're stable, document the function.  Once you have a skeleton set of tests, 
it becomes easier and easier to add to them and once they're set up, you can 
test everything and all interactions at once. (Found out a lot about differences 
in versions of gcc this way).  If you can't do XP, you absolutely need automated 
regressions tests.  If you can do XP, you still need it.

hjm


Jason E. Stewart wrote:
> ja...@op... (Jason E. Stewart) writes:
> 
> 
>>I'm concerned that my previous attempt to explain the BioAssay
>>hierarchy was less than succesful, so I thought I would take a moment
>>to go into more detail, and provide an example.
> 
> 
> In the process of writing up this document I discovered a couple
> glaring omission in the schema:
> 
> 1) There is currently no mechanism to determine which BioAssay's were
>    used to create a DBA.
> 
> 2) Any PBA could only create a *single* MBA - although this is the
>    majority case the FE Software program multiple times with different
>    parameter settings to produce multiple MBA's.
> 
> These can be handled easily by adding a BioAssayLink linking table. I
> will fix this immediately
> 
> Finaly, how to handle certain PBA situations:
> 
> 1) What if a chip is scanned multiple times at different PMT voltages,
>    one low and one high - this is relatively common.
> 2) What if a chip is hybridized and scanned, and then stripped and
>    re-hybridized and re-scanned - this is not so common.
> 
> MAGE allows for PBA's to produce other PBA through an event it calls
> the 'BioAssayTreatment'. So for the first case you can represent it by
> one PBA for the low-voltage setting that produces a second PBA for the
> high-voltage setting (assuming that is the order in which the scans
> were exectuted). For the second case you have one PBA for the first
> hyb, and then this PBA produces a second PBA for the re-hyb.
> 
> In this way there is a direct relationship in the DB about what
> hybridizations are related to each other. This makes it clear what
> events happened and in what order.
> 
> To solve this we can use the same BioAssayLink table. Most PBA's won't
> have any other PBA that produced them, but for researchers that need
> this flexibility it will be there for them without extra complexity in
> the DB.
> 
> The only added complexity we get is in *explaining* what it is and how
> to use it...
> 
> Cheers,
> jas.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle 10g. 
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev
> 

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

Re: [GeneX-dev] make configure

[Harry M. <hj...@ta...>]

This sounds good - I was going to suggest that we consolidate/clean up the 
recon/configure divergence anyway.

And making things clearly genex-related is a good idea for all kinds of reasons.

Jennifer, what's the status of matrix re: data loading?  DO you want me to 
refresh it to get these things incorporated or should we leave it alone - these 
changes are administrative and won't change any db/schema/interface details.

hjm


Jason E. Stewart wrote:
> Hey all,
> 
> It looks like a change I made to the 'Configure' script on 3/23/2004
> got lost when recon.pl was split off - I had renamed the 'readonly'
> user to 'sessions' because that is the name of the only table that
> user has access to. This wasn't obvious because I never use the
> hard-coded name of any user in the API - I always use the Perl
> Config.pm variable, e.g. $Bio::Genex::Config->{GENEX_TEST_USER}.
> 
> But it showed up when I was helping Durga debug her installer problem.
> 
> I'm proposing the following changes:
> 1) Rename recon.pl to Configure - make it the 'official' configuration
>    tool. It's had enough testing now, I think it's stable.
> 
> 2) Make the genex administrative user names clearly genex
>    specific. This is in keeping with the role names, e.g. genex_admin,
>    genex_user, etc. So instead of 'sessions' make it 'genex_sessions',
>    etc. This way our administrative users don't conflict with other
>    potential users.
> 
> 3) Instead of only a test user that has only USER level priveleges,
>    add a second user that has CURATOR priveleges as well. This came up
>    when I was adding the sample data set - I needed a user with
>    curator priveleges, so I had to give them to the 'test' user.
> 
> These changes are relatively minor, but I believe they will make long
> term maintenance simpler.
> 
> I'm asking this to the list, because it will require a 'make
> uninstall' for every installation (the old users will need to be
> deleted from the DB).
> 
> Cheers,
> jas.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle 10g. 
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev
> 

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

Re: [GeneX-dev] BioAssay's

[<ja...@op...>]

ja...@op... (Jason E. Stewart) writes:

> I'm concerned that my previous attempt to explain the BioAssay
> hierarchy was less than succesful, so I thought I would take a moment
> to go into more detail, and provide an example.

In the process of writing up this document I discovered a couple
glaring omission in the schema:

1) There is currently no mechanism to determine which BioAssay's were
   used to create a DBA.

2) Any PBA could only create a *single* MBA - although this is the
   majority case the FE Software program multiple times with different
   parameter settings to produce multiple MBA's.

These can be handled easily by adding a BioAssayLink linking table. I
will fix this immediately

Finaly, how to handle certain PBA situations:

1) What if a chip is scanned multiple times at different PMT voltages,
   one low and one high - this is relatively common.
2) What if a chip is hybridized and scanned, and then stripped and
   re-hybridized and re-scanned - this is not so common.

MAGE allows for PBA's to produce other PBA through an event it calls
the 'BioAssayTreatment'. So for the first case you can represent it by
one PBA for the low-voltage setting that produces a second PBA for the
high-voltage setting (assuming that is the order in which the scans
were exectuted). For the second case you have one PBA for the first
hyb, and then this PBA produces a second PBA for the re-hyb.

In this way there is a direct relationship in the DB about what
hybridizations are related to each other. This makes it clear what
events happened and in what order.

To solve this we can use the same BioAssayLink table. Most PBA's won't
have any other PBA that produced them, but for researchers that need
this flexibility it will be there for them without extra complexity in
the DB.

The only added complexity we get is in *explaining* what it is and how
to use it...

Cheers,
jas.