genex-dev — GeneX developers

[GeneX-dev] Status Update

[<ja...@op...>]

Hey All,

I've been slogging through building some tools to simplify adding data
processing protocols to Genex. It's taken me a bit of time to
understand how curators would want to interact with the system. But I now
have a Mason GUI that adds a new processing protocol for external
commandline apps as well as for SQL stored procedures. This tool
guides the curator through adding all the needed pieces. 

There is a second GUI that enables the curator to add additional
parameters (beyond the required ones) - I decided to keep them
seperate for now just for code re-user purposes. We may decide later
to make a single combined tool.

My next step tomorrow is to but the protocols I add through their
paces - actually using them for data processing. That will test a
number of pieces:
* the new schema changes to Parameter and Procedure
* the scratch table Float => Int conversion
* how convenient scratch views are

Tata for now,
jas.

[GeneX-dev] Schema Changes coming

[<ja...@op...>]

Hey All,

I'm testing adding data processing protocols to the DB, and I've
discovered that a couple of tables would like extra information to be
stored, e.g. Parameter and Procedure.

I'm maintaining a DB upgrade script that will non-destructively add
all the changes I'm making to a VERSION: 20040507 genex DB - which is
the version I timestamped on the DB when I froze the schema. 

More later ...
jas.

[GeneX-dev] Red Flag on ROO

[Harry M. <hj...@ta...>]

Hi All,

FYI & ITPOFD (In The Pursuit Of Full Disclosure), I've been reading up on PyUNO 
and talking to some other developers who tried to develop useful packages using 
the PyUNO approach and both of the ones whom I've etalked to have been pretty 
pessimistic about it.  I include the reply from the developer of thessalonica, 
which I've been using as a code example.  It's not clear yet whether this is a 
ROO-killer, but we should know by the end of next week.  The requirement for rpy 
does, on the face of it, look like it will fail unless we mod it to include an 
UNO interface (which might not be too noxious, if that's the only thing we need 
to do).  Alexej is moving to java to escape the python limitations, and it's a 
possibility to move ROO to java as well, altho java does make me break out in hives.

Harry

-------- Original Message --------
Subject: Re: Comments on Thessalonica, PyUNO
Date: Fri, 18 Jun 2004 21:08:08 +0400
From: Alexej Kryukov <akr...@ne...>
Reply-To: ak...@ig...
To: Harry J Mangalam <hj...@ta...>
References: <200...@ta...>

On Thursday 17 June 2004 00:00, you wrote:
>
> From what I understand, the OOo-supplied python is required because
> of linking/lib requirements for the rest of OOo and because of this,
> it's impossible to use the system python.  This also causes a number
> of external (but essential) modules to fail to work and therefore at
> least some people have indicated on the list that this is a fatal
> flaw in PyUNO.
>
> From your web page, you seem to be porting thessalonica to Java - is
> this because of the problems with PyUNO or is it due to other issues?
>
> Do you feel that the current situation (if I've described it
> correctly..?) prevents developing a large add-on in python?  

Hi Harry,

yes, everything you have described is correct and impossibility
to use an external python installation is really crucial. However,
there are several other issues which make developing large projects
with PyUNO a really challenging task, although Joerg Budishewsky
works hard to fix them. For example, have a look at this page:
http://www.openoffice.org/issues/show_bug.cgi?id=21281
Well, this problem is fixed now, but, since Python *.pyd libraries
are broken in OOo 1.1.0 and 1.1.1 which are still widely
used, this means that loading even standard python modules should
be avoided, if these modules are related with any compiled
libraries. Also you should be aware of the following issue:
http://www.openoffice.org/issues/show_bug.cgi?id=23546
(this page explains some details of the way how thessalonica
modules are implemented).

However, the most important problem is that currently there is no
easy and user friendly way to extend OOo supplied Python with new
libraries. It is impossible to distribute them in UNO packages
(like thessalonica*.uno.zip), because pkgchk assumes any *.py
file should contain a UNO component. Otherwise, an error is
reported. And even if I could install a library, it is still
difficult to import it. So, look how Thessalonica is implemented:
although it includes several *.py files, each of them contains
a separate UNO component, and interaction with these components
is always performed via UNO (not with Python). As you can see,
Java should be preferred for large projects just because it
hasn't this limitation.

Some of the problems I mentioned were recently discussed in
dev@udk, and I *hope* the pkgchk developer (Daniel Boezle) will
consider at least some of my (and Joerg's) wishes. However,
currently we have just the opposite situation, since even my
Python projects which worked fine in 1.1 are no longer functional
in the latest 680 snapshots. This is one more reason to prefer
Java: at least I am sure that any components implemented in
Java should be functional in any future releases (since OOo
itself includes many Java components, and they are always tested
by developers).

> For our
> project, we would need to have the OOo python run rpy which in turn
> calls R as a shared library as well as a number of non-standard
> libraries and modules.  it would also require quite a bit of
> interface manipulation (determining selected ranges, counting rows,
> placing output ranges, setting format characteristics, etc).
>
> Also, where is the documentation for the python API?  is it buried in
> the SDK package or is there a web site that describes it?

I'm unsure I've understood the last question. Basically, OOo API is
always the same, no matter which language do you use. You should
just know how to obtain a Service Manager object, from which all
other objects are derived. Once you have it, you can access any
functionality described in OOo SDK, and everything you have listed
above in particular. For example, determining selected ranges will
work exactly by the same way, no matter, if you use Basic, or Python,
or Java. Of course, there are language specific issues, but they
are related mainly with type conversion, as well as with defining
custom UNO components and their interaction with OOo (as I've
mentioned above, Python currently can't compete with Java at this
point, although it has some advantages in comparison with Basic).

Also, have you studied the main PyUNO page
(http://udk.openoffice.org/python/python-bridge.html)? I think it
contains almost everything you should know to start developing
your applications with PyUNO. Of course, further instructions
should be found in OOo SDK and, unfortunately, in OOo bug
tracking system ;-)

With best wishes,
Alexej



-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

[GeneX-dev] [ genex-Bugs-956693 ] using oid for pkey is breaking DB insertion code

[SourceForge.net <no...@so...>]

Bugs item #956693, was opened at 2004-05-19 06:56
Message generated for change (Comment added) made by jason_e_stewart
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=956693&group_id=16453

Category: DB Schema
Group: Genex-2
Status: Open
Resolution: None
Priority: 9
Submitted By: Jason E. Stewart (jason_e_stewart)
Assigned to: Jason E. Stewart (jason_e_stewart)
Summary: using oid for pkey is breaking DB insertion code

Initial Comment:
the change to make oid act as pkey is attempting to
make bogus insert statements. 

SQL=[[INSERT INTO GENEX_ROLE
(role_name_ont_fk,oid,other_table_fk,other_fk) VALUES
('305','641','161','637')]], DBI=[[ERROR:  column "oid"
of relation "genex_role" does not exist]]

----------------------------------------------------------------------

>Comment By: Jason E. Stewart (jason_e_stewart)
Date: 2004-06-18 01:43

Message:
Logged In: YES 
user_id=85550

This is a test comment, please ignore

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=956693&group_id=16453

Re: [GeneX-dev] Caltech on deck...

[<ja...@op...>]

Harry Mangalam <hj...@ta...> writes:

> So the short version is that GeneX should be easy to install and
> once installed, it's ready to have data thrown at it.

I just wanted to add something to this. 

Part of the default install loads a set of test data into the DB, I
would also like to add a small set of test tools that can be used on
the test data, it hasn't yet been added, but can easily be done in a
day or so.

I'll work on it now and post to the list as soon as it is working.

Cheers,
jas.

Re: [GeneX-dev] Caltech on deck...

[<ja...@op...>]

Brandon King <ki...@ca...> writes:

>     I just wanted to give you a heads up that GeneX 2.x has risen
>     towards the top of our priority list again and this time we should
>     have the time to actually put significant effort into getting it
>     running here again. This attempt will probably be made some time
>     next week. So, I thought I should ask what's the status of GeneX
>     2.x (I saw that you guys have been busy)... and find out if next
>     week is good time to try to install GeneX 2.x?

Excellent! Brandon I love you!

Your timing is perfect. We froze the schema a few weeks back, and have
been fixing the documentation and other pieces of the system,
preparing for a beta release of 2.0.

We *really* need testing of the core system. We need labs like Caltech
to load their data, and run some sample data processing tools on the
data. You have a basic familiarity of the system, plus you are willing
to roll up your sleeves and poke under the hood if things aren't
behaving the way you'd like. So your involvement would be a tremendous
boost to the project at this point.

I really need feedback on:
* how to optimize the system
* how to streamline data loading
* how to streamline hooking up external data processing and data
  analysis tools (these are adding using the Protocol mechanism)
* how to manage the WWW information interface - what things to show,
  how to handle things when there is lots and lots of information

Cheers,
jas.

Re: [GeneX-dev] Caltech on deck...

[Harry M. <hj...@ta...>]

Hi Brandon,

Good to hear from you!

IMHO, from the installation POV, it should be pretty straightforward (MUCH 
shorter & easier than before due to a simplified config file) and Jason has done 
a good job with the base schema, security, and a web-based admin layer.

The less-well-knowns at this point are the views and procedures that are layered 
onto the base schema, particularly the scratch table approach and the code and 
admin procedures to support it.  The Web-based Data loader is operational but 
needs work on smoothing the process, so we would really benefit from having 
someone complain about it.

Documentation is better than it's been for a while but it's still thin in (lots 
of) places- shout at us and we'll fill it in.

Also, while we've been doing some work on testing how well it scales, we do not 
yet have solid data on really big real data.  However, Jason (on my rec - my bad 
if it's a dog) has done a lot of work in reducing the data load and hopefully 
increasing perf by  changing most of the internals to ints vs floats and we're 
learning by the day what will give us better perf.


 From the list conversations, it may or may not be clear that we've stripped out 
the few analytical routines that were included with GeneX1 and are now moving to 
  2 main interfaces - a mason-based web interface for admin, data-loading, and 
high-level data-browsing and an OpenOffice interface for queries using the ODBC 
Data Source connector (native postgres in OOo2, tho I'm not sure that it'll be 
a huge improvement - the ODBC connector is pretty slick) and query tool.  this 
will allow users to pull data into the OOo spreadsheet (Calc) and use std 
spreadsheet tools to manipulate the data.

Hrishi and Todd (2 grads at GMU) and I are also working thru the process of 
working R (and therefore the Bioconductor tools) into the OOo environment via 
the use of the PyUNO API and rpy (provides a python interface to the R shared 
lib, which can make use of any package that R can load).  Overall, its not 
tricky (theoretically) but the devil is in the details and there are some issues 
with the way python is included with pyuno in OOo 1.1.1 and the way the external 
packages are to be included in 2.0.  We're slowly working thru all the details 
and I'm still mostly optimistic that this can be done - we should have some kind 
of end-to-end proof of concept done by next week or know why we can't do it.. 
If things don't  fall apart completely in the new/2 PyUno SDK, it's a really 
nice way to include external apps as you can build some very nice interfaces 
with OOo's internal GUI builder and patch them to your python code pretty easily 
(albeit with StarBasic glue).  If you're interested in this approach I can 
forward some email about it that didn't make it to the list.

So the short version is that GeneX should be easy to install  and once 
installed, it's ready to have data thrown at it.

hjm

Brandon King wrote:
> Hi Jason & Friends,
>    I just wanted to give you a heads up that GeneX 2.x has risen towards 
> the top of our priority list again and this time we should have the time 
> to actually put significant effort into getting it running here again. 
> This attempt will probably be made some time next week. So, I thought I 
> should ask what's the status of GeneX 2.x (I saw that you guys have been 
> busy)... and find out if next week is good time to try to install GeneX 
> 2.x?
> 
> -Brandon King
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference
> Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer
> Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA
> REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev
> 

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

[GeneX-dev] Caltech on deck...

[Brandon K. <ki...@ca...>]

Hi Jason & Friends,
    I just wanted to give you a heads up that GeneX 2.x has risen 
towards the top of our priority list again and this time we should have 
the time to actually put significant effort into getting it running here 
again. This attempt will probably be made some time next week. So, I 
thought I should ask what's the status of GeneX 2.x (I saw that you guys 
have been busy)... and find out if next week is good time to try to 
install GeneX 2.x?

-Brandon King

[GeneX-dev] New WWW site online

[<ja...@op...>]

Hey All,

Harry and I have updated the SF site a bit. I've made the major
architectural changes, and now Harry can add in the content changes
he's been working on.

We'll also be making the appearance a lot easier to configure and
maintain by using a site-wide stylesheet.

Cheers,
jas.

[GeneX-dev] Maypole

[<ja...@op...>]

Hey All,

An FYI:

I just found out that the Perl Foundation gave Perl hacker
extraordinaire Simon Cousins a one month grant to write Maypole, a
system for developing web applications in Perl:

  http://www.perl.com/pub/a/2004/04/29/maypole.html

It was a very interesting article. Maypole does many similar things to
how the genex API is created, but in many ways more
elegantly. However, they have a very simplistic security model, so we
couldn't use it for Genex.

I also read about Class::DBI which is used by Maypole:

  http://www.perl.com/pub/a/2002/11/27/classdbi.html

It is a very nice, simple way to put an OO API on top of a relational
DB. 

Cheers,
jas.

[GeneX-dev] RE: scratch table and filters

[Olga B. <ob...@vt...>]

Harry and all,

I didn't follow genex development in the last year and actually unsubscribed 
from the list -- so I need more time and info to get back on track. Are you 
having a performance problem when too many users accessing the same (hopefully 
read-only) table? also how the scratch tables are implemented -- are they 
created for the session only or on a more permanent basis? Do you do any 
parallel processing?

Olga
>===== Original Message From hj...@ta... =====
>Hi Olga & All,
>
>I would appreciate any input on this.  I have talked to some other people 
about
>this and have gotten some fresh insight on it. Jason has disappeared 
(hopefully
>only temporarily) from the scene, so I haven't been able to get his input, 
but I
>also realized talking to Jennifer that I misunderstood some of the design
>rationale.  She agreed that it made GeneX a very difficult beast to admin for 
a
>completely open system, but indicated that she considered this aspect to be
>useful for a very restricted set of users - say a department or <10 users
>anyway.  In this light, I can see how it might be more useful, maintaining an
>individual workspace that could be written back into the DB for later recall.
>
>What I couldn't see was the use of allowing this as each person's workspace
>would be quite different and therefore designing queries against this scratch
>table would be impossible.
>
>I therefore see GeneX as being implemented on 2 levels:
>
>1 - the base level that provides a schema for storage of gene expression 
data.
>  We can provide queries against this DB so that multiple people can use it
>simultaneously to pull data from it.
>
>2 - a higher level that allows a few users to customize it and populate one 
(or
>a few) scratch tables on a completely custom basis for their own work.  We 
can
>supply queries against the std DB, but if you populate a scratch table it 
will
>be your own responsibility to design and implement queries against it.
>
>Jennifer, do I have that right?
>
>Harry
>
>
>Olga Brazhnik wrote:
>> Sorry -- I was away from the office for quite a while. I'll read through 
this
>> and get back to you ASAP.
>> Olga
>>
>>
>>>===== Original Message From hj...@ta... =====
>>>Hi All, especially Olga - we'd love your input on this.
>>>
>>>Jason E. Stewart wrote:
>>>
>>>I wrote a very long detailed answer to this, and I was wondering why I
>>>hadn't heard a response from anyone.
>>>
>>>Then I went digging in my sent email box and I couldn't find my
>>>reply... Arrghh!!!
>>>
>>>that sounds sooo familiar ;).  My UPS buggered up a while ago and while I
>>>haven't had any linux-related crashed that resulted in data loss (thanks to
>>>reiserfs, my journaling file system of choice), Mozilla DOES crash every 
once
>>
>> in
>>
>>>a while and every once in a while it takes all connected stuff with it
>>>(bookmarks, unsent mail, address books, etc).  So you have my sympathy.
>>>
>>>
>>>
>>>Ok, so I'll write the brief stuff first...
>>>
>>>Harry J Mangalam <hj...@ta...> writes:
>>>
>>>
>>>
>>>>3 - VACUUM frequently
>>>
>>>
>>>Harry, I discovered that Pg 7.4 has an auto-vacuum facility that
>>>handles it all based on an internal algorithm. I think we should turn
>>>it on.
>>>
>>>Good idea.  I didn't know that.
>>>
>>>
>>>>4 - use the right kind of joins to pre-filter data before applying other
>>>>criteria.
>>>
>>>
>>>Can you give an example of this type of join?
>>>
>>>Not offhand, but this is something that I picked up in picking thru the
>>
>> output
>>>from the EXPLAIN (and EXPLAIN ANALYZE).  The PG query planner can do some
>> decent
>>
>>>optimization but  if you help it by prefiltering data that will have to
>>
>> undergo
>>
>>>harder, longer comparisons.  for example (hmm - I guess I do have an
>>
>> example),
>>
>>>if you're going to do an integer comparison and a string search (and they
>>
>> both
>>
>>>have to be true) then obviously doing the integer cmp 1st will and the 
search
>>>second will be much more efficient than the reverse, especially if the
>>
>> integer
>>
>>>cmp can be otherwise optimized via an index and/or pk-fk relationship.
>>>
>>>
>>>>These points will result in most queries being instantaneous, and if
>>
>> they're
>>
>>>>not instantaneous, they will tend to take a LONG time, especially with 
lots
>>>>of rows.
>>>
>>>Maybe 'most queries being instantaneous' is overstating things...  If 
they're
>>>integer-based, they could be very fast, if they're complex queries,
>>
>> especially
>>
>>>if they're nested loops, they'll tend to be very slow so we'll have to
>>
>> monitor
>>
>>>the system closely to prevent the whole thing from slowing to a halt,
>>
>> probably
>>
>>>by LIMITing the amount of time that a query can take without some kind of
>>
>> oversight.
>>
>>>
>>>instantaneous is good for the common cases. what about copying user
>>>data to a temporary table if lots of stuff is going to be done? Won't
>>>this decrease the time needed?
>>>
>>>I don't see how this is the case (tho I don't know if it won't be the case
>>>either).  If the question is: "If you copy all the data for a query to one
>>>table, will the query be faster?", the answer will probably be yes, for 
more
>>>than one query, but will there be more than one query on this data?  And if
>>
>> 20
>>
>>>users are querying simultaneously, is this a good idea? (to have 20 scratch
>>>tables each of 20% the size of the total DB?
>>>
>>>And can this even be done?  If there are 20 simultaneous users, don't there
>>
>> need
>>
>>>to be 20 different scratch tables so the data doesn't get
>>
>> overwritten/corrupted?
>>
>>>Are there any examples in the large db arena (amazon/peoplesoft/oracle) 
where
>>>people do this?
>>>
>>>Or should we be optimizing things so that the queries operate on the 
existing
>>>tables and only /write/ data to the scratch table? where the DB system can
>>>assist in writing this back into the DB?
>>>
>>>
>>>If there is a 20 user genex system, no point in slowing down every
>>>query by a factor of 20. Just a thought.
>>>
>>>Cheers,
>>>jas.
>>>
>>>
>>>--
>>>Cheers, Harry
>>>Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
>>>            <<plain text preferred>>
>>
>>
>>
>
>--
>Cheers, Harry
>Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
>             <<plain text preferred>>

[GeneX-dev] Aquafold -new RDBMS tool - free, not opensource

[Harry M. <hj...@ta...>]

looks splashy, and works with PG, but still unsure of how well it works:
free for personal/academic use - I think GeneXqualifies.

http://www.aquafold.com/index.html


Also something like it from theKompany - Data Architect:
http://thekompany.com/products/dataarchitect/

which has a time-limited demo and sells for $60 otherwise.
-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

[GeneX-dev] Re: scratch table and filters

[Harry M. <hj...@ta...>]

Hi Olga & All,

I would appreciate any input on this.  I have talked to some other people about 
this and have gotten some fresh insight on it. Jason has disappeared (hopefully 
only temporarily) from the scene, so I haven't been able to get his input, but I 
also realized talking to Jennifer that I misunderstood some of the design 
rationale.  She agreed that it made GeneX a very difficult beast to admin for a 
completely open system, but indicated that she considered this aspect to be 
useful for a very restricted set of users - say a department or <10 users 
anyway.  In this light, I can see how it might be more useful, maintaining an 
individual workspace that could be written back into the DB for later recall.

What I couldn't see was the use of allowing this as each person's workspace 
would be quite different and therefore designing queries against this scratch 
table would be impossible.

I therefore see GeneX as being implemented on 2 levels:

1 - the base level that provides a schema for storage of gene expression data. 
  We can provide queries against this DB so that multiple people can use it 
simultaneously to pull data from it.

2 - a higher level that allows a few users to customize it and populate one (or 
a few) scratch tables on a completely custom basis for their own work.  We can 
supply queries against the std DB, but if you populate a scratch table it will 
be your own responsibility to design and implement queries against it.

Jennifer, do I have that right?

Harry


Olga Brazhnik wrote:
> Sorry -- I was away from the office for quite a while. I'll read through this 
> and get back to you ASAP.
> Olga
> 
> 
>>===== Original Message From hj...@ta... =====
>>Hi All, especially Olga - we'd love your input on this.
>>
>>Jason E. Stewart wrote:
>>
>>>I wrote a very long detailed answer to this, and I was wondering why I
>>>hadn't heard a response from anyone.
>>>
>>>Then I went digging in my sent email box and I couldn't find my
>>>reply... Arrghh!!!
>>
>>that sounds sooo familiar ;).  My UPS buggered up a while ago and while I
>>haven't had any linux-related crashed that resulted in data loss (thanks to
>>reiserfs, my journaling file system of choice), Mozilla DOES crash every once 
> 
> in
> 
>>a while and every once in a while it takes all connected stuff with it
>>(bookmarks, unsent mail, address books, etc).  So you have my sympathy.
>>
>>
>>
>>>Ok, so I'll write the brief stuff first...
>>>
>>>Harry J Mangalam <hj...@ta...> writes:
>>>
>>>
>>>
>>>>3 - VACUUM frequently
>>>
>>>
>>>Harry, I discovered that Pg 7.4 has an auto-vacuum facility that
>>>handles it all based on an internal algorithm. I think we should turn
>>>it on.
>>
>>Good idea.  I didn't know that.
>>
>>
>>>>4 - use the right kind of joins to pre-filter data before applying other
>>>>criteria.
>>>
>>>
>>>Can you give an example of this type of join?
>>
>>Not offhand, but this is something that I picked up in picking thru the 
> 
> output
>>from the EXPLAIN (and EXPLAIN ANALYZE).  The PG query planner can do some 
> decent
> 
>>optimization but  if you help it by prefiltering data that will have to 
> 
> undergo
> 
>>harder, longer comparisons.  for example (hmm - I guess I do have an 
> 
> example),
> 
>>if you're going to do an integer comparison and a string search (and they 
> 
> both
> 
>>have to be true) then obviously doing the integer cmp 1st will and the search
>>second will be much more efficient than the reverse, especially if the 
> 
> integer
> 
>>cmp can be otherwise optimized via an index and/or pk-fk relationship.
>>
>>
>>>>These points will result in most queries being instantaneous, and if 
> 
> they're
> 
>>>>not instantaneous, they will tend to take a LONG time, especially with lots
>>>>of rows.
>>
>>Maybe 'most queries being instantaneous' is overstating things...  If they're
>>integer-based, they could be very fast, if they're complex queries, 
> 
> especially
> 
>>if they're nested loops, they'll tend to be very slow so we'll have to 
> 
> monitor
> 
>>the system closely to prevent the whole thing from slowing to a halt, 
> 
> probably
> 
>>by LIMITing the amount of time that a query can take without some kind of 
> 
> oversight.
> 
>>
>>>instantaneous is good for the common cases. what about copying user
>>>data to a temporary table if lots of stuff is going to be done? Won't
>>>this decrease the time needed?
>>
>>I don't see how this is the case (tho I don't know if it won't be the case
>>either).  If the question is: "If you copy all the data for a query to one
>>table, will the query be faster?", the answer will probably be yes, for more
>>than one query, but will there be more than one query on this data?  And if 
> 
> 20
> 
>>users are querying simultaneously, is this a good idea? (to have 20 scratch
>>tables each of 20% the size of the total DB?
>>
>>And can this even be done?  If there are 20 simultaneous users, don't there 
> 
> need
> 
>>to be 20 different scratch tables so the data doesn't get 
> 
> overwritten/corrupted?
> 
>>Are there any examples in the large db arena (amazon/peoplesoft/oracle) where
>>people do this?
>>
>>Or should we be optimizing things so that the queries operate on the existing
>>tables and only /write/ data to the scratch table? where the DB system can
>>assist in writing this back into the DB?
>>
>>
>>>If there is a 20 user genex system, no point in slowing down every
>>>query by a factor of 20. Just a thought.
>>>
>>>Cheers,
>>>jas.
>>>
>>
>>--
>>Cheers, Harry
>>Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
>>            <<plain text preferred>>
> 
> 
> 

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

Re: [GeneX-dev] Beta Release Test Plan

[Harry M. <hj...@ta...>]

HI Jason,

I could have sworn I replied to this - I even remembered the phrases I used; but 
I can't find it in my outbox, so it may have died along with a mozilla crash 
that wiped out all my bookmarks as well.  Sorry.



Jason E. Stewart wrote:
> Hey All,
> 
> Here's a draft of a plan for testing genex before making a release.
> 
> 1) Test that fresh install runs without errors - i.e. 
>   - make uninstall
>   - make configure
>   - make install

good.

> 2) Test Mason components
>   - which should be tested?

preferably all of them, but minimally, the  ones that must be used to load and 
export data.

> 3) Test automateable perl scripts
>   - which should be tested?

If they're automatable, they should all be tested, no?

> 4) Test 'make test' on all API modules runs w/o errors
>   - Bio-Genex
>   - Bio-MAGE
>   - Class-ObjectTemplate
>   - Class-ObjectTemplate-DB

good.

> 5) All required docs up-to-date
>   - which are required



The current docs are:

User level
----------
ArrayDesignCreation-HOWTO.html      [U]     2003-10-31 JES
DataLoading-HOWTO.html              [utd]   2004-04-04 JES
GeneSpring_GeneX.database           [utd]   2003-02-13 HJM *
GeneSpring.notes                    [utd]   2003-02-13 HJM *
GeneX-ODBC.HOWTO                    [utd]   2004-05-26 HJM
GeneX-Schema.pdf                    [utd]   2004-05-04 HJM
HOWTO_Load_QuantArray_Data.txt      [U]     2003-02-13 HJM
    (should be merged into DataLoading-HOWTO.html)
howto-new_genex2_file_format.txt    [U]     2004-05-05 BK, hjm
    (should be merged into ArrayDesignCreation-HOWTO.html)
mba-explanation.html                [utd]   2004-05-10 JES, hjm
miame_glossary.html                 [utd]   2004-05-10 HJM
ReleaseNotes.txt                    [U]     2001-11-21 JES
Sample-HOWTO.html                   [U]     2003-10-31 JES
scratch.tables.and.filters.txt      [U]     2004-05-26 HJM
Security.txt                        [utd]   2004-04-06 JES

[U]   - needs to be updated
[utd] - up to date
* dated but up to date as far as we know


XML sources
-----------
Sample-HOWTO.xml
ArrayDesignCreation-HOWTO.xml
DataLoading-HOWTO.xml
DataLoading-HOWTO.xml.in

Dev level
---------
Using.CPAN                          [utd]   2004-05-25 HJM
GeneX_DB_tuning.txt                 [utd]   2004-05-26 HJM
genex-ents.xml                      [?]     2004-04-06 JES
docbook2html.pl (script)
codingGuideline.txt                 [U]     2000-14-04 JES,JZ
Contributing                        [U]     2000-14-04 JES,JZ
Cutting_a_Release.notes             Obsolete
Darwin_HOWTO_Template.xml           Docbook template?



> 6) External data analysis
>   - should OO.o be included in beta?

No, not at this time if you're talking about the integration of R/BioC, but the 
docs that describe the ODBC connection should definitely be pumped to show how 
things work together.

> 7) Other? WWW site?

Yes, I'm working (slowly) on getting this sucker up to date and in svn.

> For part 1) perhaps we should attempt some optional installation
> parameters as well, such as: alternate installation locations for the
> three different directories (html, cgi, bin), tcp/ip support, others?

I'd say no - keep it on the default paths for now - if people want to install on 
alterative paths, we'll help them / debug as this comes up.

> For part 3), this can be automated - and most of it is tested in
> gendb.pl at installation time already.

OK.

> For part 4) only Bio-Genex and Class-ObjectTemplate-DB change, and I
> run 'make test' on them whenever I remember (although I've been rather
> bad with Class-ObjectTemplate-DB and not written any useful regression
> tests in a long time).

OK - maybe add a make test to the Makefile and have it run automatically when it 
gets made?

> For part 5) we should look at the SF task tracker and prioritize the
> documents: all 7,8, and 9 get included in the beta, and 5 and 6 for
> the full release.

OK - I'll go to it and start checking things off.
> 
> That leaves part 2). We should test:
> - insert DB User
> - change password
> - set user preferences
> - all required docs available
> - Full data load path:
>   * insert Experiment (user)
>   * insert Biomaterial (user)
>   * load ArrayDesign (curator)
>   * insert ArrayManufacture (user)
>   * insert Array (user)
>   * load MBA Data (user) - also adds MBA and PBA
>   * add Image to PBA (user)
>   * copy MBA Data to Scratch (user)
>   * run processing protocol on Scratch data
>   * add scratch view (curator)???
>   * add processing protocol (curator)???
>   * annotate BiomaterialCharacteristics (user)???
> - export DBA/MBA data to tab file
> - other???

OK - this will be a lot of work, but it WILL be required to use teh DB, so WE'D 
better do it 1st to make sure it works end to end.  AND we need an end-to-end 
tutorial for it as well.  After I finish with the other docss, I can work on 
this and then pass  my stuff on to JWW to repeat it and add her comments.



-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

Re: [GeneX-dev] small buglets / questions

[Harry M. <hj...@ta...>]

>>ALSO, many of the URLs to local docs are written in (data-loading.html, for 
>>example) as: <a 
>>href="http://localhost/genex/mason/login/workspace/edit/biomaterial-edit.html">
>>blah </a>
>>
>>This point's to the user's machine and will usually fail (unless she is 
>>accessing a genex db on that machine).  How about citing local installations 
>>as relative URLs?
>>
>><a href="genex/mason/login/workspace/edit/biomaterial-edit.html"> blah </a>
> 
> 
> Ah, good point, can you change them? Since data-loading.html is at:
> 
>   genex/mason/nologin/docs/data-loading.html
> 
> The relative URL should be something more like:
> 
>   <a href="../../login/workspace/edit/biomaterial-edit.html">
> 
> shouldn't it?
> 
> The other approach is to put
> 
>   "%%MASON_WORKSPACE_URL%%/edit/biomaterial-edit.html"
> 
> in DataLoading-HOWTO.xml and make it DataLoading-HOWTO.xml.in so that
> subsitution will work. The clean way is would be the latter, but I'm
> cool with either.

OK - done.


-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta...
             <<plain text preferred>>

[GeneX-dev] Tested on genex2 - no install errors

[<ja...@op...>]

Hey All,

All my changes are in, so it should work on a machine near you.

A simple: 'sudo make install_files' should get you all you need => no
schema changes.

Cheers,
jas.

[GeneX-dev] [ genex-Bugs-960106 ] update on experimentlink-edit.html causes mason dump

[SourceForge.net <no...@so...>]

Bugs item #960106, was opened at 2004-05-25 06:51
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=960106&group_id=16453

Category: Mason GUI
Group: Genex-2
Status: Open
Resolution: None
Priority: 6
Submitted By: Jason E. Stewart (jason_e_stewart)
Assigned to: Jason E. Stewart (jason_e_stewart)
Summary: update on experimentlink-edit.html causes mason dump

Initial Comment:
Trying to edit an ExperimentLink entry from a user with
permission to modify causes a Mason dump

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=960106&group_id=16453

[GeneX-dev] oids

[<ja...@op...>]

Hey All,

Just an FYI.

I while back we had to make the Postgres administrative collumn,
'oid', available to all the views. As I understand it, 'oid' is a
funny beast that is used internally by Postgres as a unique primary
key for *every* row of *every* table in *every* DB in the Postgres
installation. 

Well, that got me thinking - The Bio-Genex API used to have some
trouble with tables that lacked primary keys, there were certain
things that were just too complicated for me to sit down and figure
out how to do it without a pkey, like supporting the update_db() and
delete_db() methods for linking tables. That makes it a bit of a pain
for certain regular things that a developer would like to do.

Well it turns out that by using oid's as the primary key value for
those tables without primary keys, we get just the functionality we
need. So now tables without pkeys have all of the same API methods as
tables with pkeys, and the Mason GUI now supports deleting and
modifying entries in linking tables!

Cheers,
jas.

PS. Why don't we use the oid column as our primary key for all our
tables and just be done with it, you may ask? From what I've read, it
is heavily discouraged to use them for anything that needs to be
stable across DB backups and reloads, because you have to do special
things to get them preserved, plus, the Pg admins are warning that in
future version the 'oid' column may disappear.

Re: [GeneX-dev] small buglets / questions

[<ja...@op...>]

Harry J Mangalam <hj...@ta...> writes:

> As of 5.24.04, the files that generated the errors I previously mentioned:
> !! System Error: No such file or directory @ line: 286  (cp 
> Perl/scripts/scanner-insert.pl /usr/local/genex/bin)
>
> !! System Error: No such file or directory @ line: 286  (cp 
> Perl/scripts/software-insert.pl /usr/local/genex/bin)
>
> !! System Error: No such file or directory @ line: 286  (cp 
> Perl/scripts/species-insert.pl /usr/local/genex/bin)
>
> !! System Error: No such file or directory @ line: 286  (cp 
> Perl/scripts/spotter-insert.pl /usr/local/genex/bin)
>
> !! System Error: No such file or directory @ line: 286  (cp 
> Perl/Bio-Genex/html/ExperimentFactors.html /var/www/genex/mason/nologin/docs/perl-api)
>
> !! System Error: No such file or directory @ line: 286  (cp 
> Perl/Bio-Genex/html/ExternalDatabase.html /var/www/genex/mason/nologin/docs/perl-api)
>
> So these refs are still in the MANIFEST - I can remove them, unless Jason 
> already has done so and has yet to commit them.

Sorry, done locally, not committed yet.

> ALSO, many of the URLs to local docs are written in (data-loading.html, for 
> example) as: <a 
> href="http://localhost/genex/mason/login/workspace/edit/biomaterial-edit.html">
> blah </a>
>
> This point's to the user's machine and will usually fail (unless she is 
> accessing a genex db on that machine).  How about citing local installations 
> as relative URLs?
>
> <a href="genex/mason/login/workspace/edit/biomaterial-edit.html"> blah </a>

Ah, good point, can you change them? Since data-loading.html is at:

  genex/mason/nologin/docs/data-loading.html

The relative URL should be something more like:

  <a href="../../login/workspace/edit/biomaterial-edit.html">

shouldn't it?

The other approach is to put

  "%%MASON_WORKSPACE_URL%%/edit/biomaterial-edit.html"

in DataLoading-HOWTO.xml and make it DataLoading-HOWTO.xml.in so that
subsitution will work. The clean way is would be the latter, but I'm
cool with either.

Cheers,
jas.

[GeneX-dev] small buglets / questions

[Harry J M. <hj...@ta...>]

As of 5.24.04, the files that generated the errors I previously mentioned:
!! System Error: No such file or directory @ line: 286  (cp 
Perl/scripts/scanner-insert.pl /usr/local/genex/bin)

!! System Error: No such file or directory @ line: 286  (cp 
Perl/scripts/software-insert.pl /usr/local/genex/bin)

!! System Error: No such file or directory @ line: 286  (cp 
Perl/scripts/species-insert.pl /usr/local/genex/bin)

!! System Error: No such file or directory @ line: 286  (cp 
Perl/scripts/spotter-insert.pl /usr/local/genex/bin)

!! System Error: No such file or directory @ line: 286  (cp 
Perl/Bio-Genex/html/ExperimentFactors.html /var/www/genex/mason/nologin/docs/perl-api)

!! System Error: No such file or directory @ line: 286  (cp 
Perl/Bio-Genex/html/ExternalDatabase.html /var/www/genex/mason/nologin/docs/perl-api)

So these refs are still in the MANIFEST - I can remove them, unless Jason 
already has done so and has yet to commit them.

ALSO, many of the URLs to local docs are written in (data-loading.html, for 
example) as: <a 
href="http://localhost/genex/mason/login/workspace/edit/biomaterial-edit.html">
blah </a>

This point's to the user's machine and will usually fail (unless she is 
accessing a genex db on that machine).  How about citing local installations 
as relative URLs?

<a href="genex/mason/login/workspace/edit/biomaterial-edit.html"> blah </a>

or is there a reason not to do this?

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta... 
            <<plain text preferred>>

Re: [GeneX-dev] Beta Release Test Plan

[<ja...@op...>]

ja...@op... (Jason E. Stewart) writes:

> Hey All,
>
> Here's a draft of a plan for testing genex before making a release.

Hey everyone,

I haven't heard any feedback on this yet. 

I'm anxious to get a beta-release, and I feel we have to define *what*
we are releasing (which pieces work and how they work), and how we
will test the pieces in order to have any chance of releasing
something worth outside peoples time to install and test.

Cheers,
jas.

> 1) Test that fresh install runs without errors - i.e. 
>   - make uninstall
>   - make configure
>   - make install
>
> 2) Test Mason components
>   - which should be tested?
>
> 3) Test automateable perl scripts
>   - which should be tested?
>
> 4) Test 'make test' on all API modules runs w/o errors
>   - Bio-Genex
>   - Bio-MAGE
>   - Class-ObjectTemplate
>   - Class-ObjectTemplate-DB
>
> 5) All required docs up-to-date
>   - which are required?
>
> 6) External data analysis
>   - should OO.o be included in beta?
>
> 7) Other? WWW site?
>
> For part 1) perhaps we should attempt some optional installation
> parameters as well, such as: alternate installation locations for the
> three different directories (html, cgi, bin), tcp/ip support, others?
>
> For part 3), this can be automated - and most of it is tested in
> gendb.pl at installation time already.
>
> For part 4) only Bio-Genex and Class-ObjectTemplate-DB change, and I
> run 'make test' on them whenever I remember (although I've been rather
> bad with Class-ObjectTemplate-DB and not written any useful regression
> tests in a long time).
>
> For part 5) we should look at the SF task tracker and prioritize the
> documents: all 7,8, and 9 get included in the beta, and 5 and 6 for
> the full release.
>
> That leaves part 2). We should test:
> - insert DB User
> - change password
> - set user preferences
> - all required docs available
> - Full data load path:
>   * insert Experiment (user)
>   * insert Biomaterial (user)
>   * load ArrayDesign (curator)
>   * insert ArrayManufacture (user)
>   * insert Array (user)
>   * load MBA Data (user) - also adds MBA and PBA
>   * add Image to PBA (user)
>   * copy MBA Data to Scratch (user)
>   * run processing protocol on Scratch data
>   * add scratch view (curator)???
>   * add processing protocol (curator)???
>   * annotate BiomaterialCharacteristics (user)???
> - export DBA/MBA data to tab file
> - other???
>
> If this is totally wackoo, let me know - but we need some checklist so
> that we can get a beta release out the door REAL SOON!!!
>
> Cheers,
> jas.

Re: [GeneX-dev] libapache-session-perl

[<ja...@op...>]

Harry J Mangalam <hj...@ta...> writes:

> Hi Jason - does the Debian pkg  libapache-session-perl provide teh same 
> functionality as  Apache::Session::Postgres?

Yes! I somehow overlooked that it was already available. In fact the
package was already installed on my system.

Apache::Session::Postgres is one small part of the whole
Apache::Session module.

Cheers,
jas.

[GeneX-dev] libapache-session-perl

[Harry J M. <hj...@ta...>]

Hi Jason - does the Debian pkg  libapache-session-perl provide teh same 
functionality as  Apache::Session::Postgres?

I can't tell from the pkg description..


When installing on another machine that's less genexified than my lappie, I 
got:

       Found Required Perl Module: DBD::Pg
        Found Required Perl Module: DBI
        Found Required Perl Module: Term::ReadKey

         #### Error ####

The perl interpreter you told me to use, '/usr/bin/perl',
Is missing a module or has the following problem:

  Could not load module: Apache::Session::Postgres:

If we could install the most things via apt-get, it will be easier
-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta... 
            <<plain text preferred>>

Re: [GeneX-dev] latest genex builds and installs ok on bodi, bonk

[<ja...@op...>]

Harry J Mangalam <hj...@ta...> writes:

> ..(my 2 machines here).
>
> There was a confusion about installing the Apache::SubProcess module on one 
> machine, but it was eventually ironed out - I missed doing things in order 
> and that may have been one of the problems.  Otherwise, if I had actually 
> followed the instructions, things went smoothly with the exception
> of:

Great!

>  1042 $ cat install-errors.txt
>
> !! System Error: Bad file descriptor @ line: 286  (cp 
> Perl/scripts/scanner-insert.pl /usr/local/genex/bin)
>
> !! System Error: Bad file descriptor @ line: 286  (cp 
> Perl/scripts/software-insert.pl /usr/local/genex/bin)
>
> !! System Error: Bad file descriptor @ line: 286  (cp 
> Perl/scripts/species-insert.pl /usr/local/genex/bin)
>
> !! System Error: Bad file descriptor @ line: 286  (cp 
> Perl/scripts/spotter-insert.pl /usr/local/genex/bin)
>
> !! System Error: No such file or directory @ line: 286  (cp 
> Perl/Bio-Genex/html/ExperimentFactors.html /var/www/genex/mason/nologin/docs/perl-api)
>
> !! System Error: No such file or directory @ line: 286  (cp 
> Perl/Bio-Genex/html/ExternalDatabase.html /var/www/genex/mason/nologin/docs/perl-api)

My bad. fixed locally - I deleted these files on my machine and I knew
they were in the MANIFEST, but I still didn't correct it before
committing the work.

jas.

[GeneX-dev] latest genex builds and installs ok on bodi, bonk

[Harry J M. <hj...@ta...>]

..(my 2 machines here).

There was a confusion about installing the Apache::SubProcess module on one 
machine, but it was eventually ironed out - I missed doing things in order 
and that may have been one of the problems.  Otherwise, if I had actually 
followed the instructions, things went smoothly with the exception of:

 1042 $ cat install-errors.txt

!! System Error: Bad file descriptor @ line: 286  (cp 
Perl/scripts/scanner-insert.pl /usr/local/genex/bin)

!! System Error: Bad file descriptor @ line: 286  (cp 
Perl/scripts/software-insert.pl /usr/local/genex/bin)

!! System Error: Bad file descriptor @ line: 286  (cp 
Perl/scripts/species-insert.pl /usr/local/genex/bin)

!! System Error: Bad file descriptor @ line: 286  (cp 
Perl/scripts/spotter-insert.pl /usr/local/genex/bin)

!! System Error: No such file or directory @ line: 286  (cp 
Perl/Bio-Genex/html/ExperimentFactors.html /var/www/genex/mason/nologin/docs/perl-api)

!! System Error: No such file or directory @ line: 286  (cp 
Perl/Bio-Genex/html/ExternalDatabase.html /var/www/genex/mason/nologin/docs/perl-api)


enteered as a SF bug report as well.

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta... 
            <<plain text preferred>>