genex-dev — GeneX developers

[GeneX-dev] types of users in genex (was: [#1] Re: task differentiation)

[<ja...@op...>]

Harry Mangalam <hj...@ta...> writes:

> Didn't we decide to eliminate curators?  I thought that we were
> going to settle on admins and users.

Not yet - there was a role called PI that existed for a while - that
was what got eliminated. I really think keeping curator is a good
idea. Admins have access to changing passwords and priveleges, and to
all data in the DB. Curators have access to the curation tables of
public data, but not to any private data.

Seems like an important difference to me, but maybe no one will ever
use it.

Cheers,
jas.

[GeneX-dev] Re: How to create AD file for Affymetrix data

[<ja...@op...>]

Hrishikesh Deshmukh <hde...@gm...> writes:

> One can use Data Transfer Tool to get data out of GCOS data as .CAB
> file, but the data which went in GCOS was MAS 5.0
>
> Now i have to figure how to convert GCOS data to MAS 5.0 data!

Hey Hrishi,

We don't have a MAS5 data loader module - only a .CEL file module. I'm
happy to write a MAS5 module - I just need an example data file.

Cheers,
jas.

[GeneX-dev] Re: [Fwd: Fwd: RE:]

[<ja...@op...>]

Hrishikesh Deshmukh <hde...@gm...> writes:

> Hi All,
>
>
> I thought this might be useful information when writing parsers for
> Affymetrix (.cel,.cdf,etc) files.
>
>> Subject: RE: Date: Mon, 15 Nov 2004 11:54:57 -0500
>> See Affy file format:
>>
> http://www.affymetrix.com/support/developer/AffxFileFormats.ZIP

Hey Hrishi,

Thanks, I'll keep this around.

Cheers,
jas.

[GeneX-dev] Re: Interesting Link!

[<ja...@op...>]

Hrishikesh Deshmukh <hde...@gm...> writes:

> http://dot.ped.med.umich.edu:2000/ourimage/pub/shared/Affymethods.html
> http://dot.ped.med.umich.edu:2000/ourimage/rdk/readaffy.html

Thanks Hrishi. When I get to a high bandwidth link soon, I'll grab the
software (it's around ~20Mb).

Cheers,
jas.

Re: [GeneX-dev] Problem loading cdf file

[<ja...@op...>]

Hrishikesh Deshmukh <hde...@gm...> writes:

> For MAGE-ML creation the docs point to ADCreation How-To and here is
> what i get,i have already sent you the output for fix.pl script
> which i sent. Kindly adivse.

Hrishi,

Did you figure this out? Your later emails seem to indicate this is
not an issue for you. If your still not clear let me know.

MAGE-ML ArrayDesign files are no longer required for Affy data - .CDF
files are all that is needed.

Cheers,
jas.

Re: [GeneX-dev] Problem loading cdf file

[<ja...@op...>]

Hrishikesh Deshmukh <hde...@gm...> writes:

> Lonnnnnnnnngggg Timeeeeeeeeee

1200 secs = 20 minutes - hah!! You haven't seen a long time until you
try loading a 400,000 line .CEL trying to use INSERT instead of COPY!!

I decided to leave .CDF insertion along, and let it just user INSERT
without dropping indices or using COPY - but if people think that 20
mins is too long, we can change it.

Cheers,
jas.

Re: [GeneX-dev] Problem loading cdf file

[Hrishikesh D. <hde...@gm...>]

Jason,

For MAGE-ML creation the docs point to ADCreation How-To and here is 
what i get,i have already sent you the output for fix.pl script which i 
sent. Kindly adivse.

Thanks,
Hrishi

deshmuk@genex2:/usr/local/genex/uploads/genex$ perl 
/usr/local/genex/bin/tab2AD.pl --in HG_U95Av2-AD.txt --out 
HG_U95Av2-AD.xml --ad_identifier=genex.org/ArrayDesign/2003-10-31/Affy-001
Use of uninitialized value in concatenation (.) or string at 
/usr/local/genex/bin/tab2AD.pl line 134, <IN> line 1.
Found  features
Parsing took: 0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)
Use of uninitialized value in concatenation (.) or string at 
/usr/local/genex/bin/tab2AD.pl line 138, <IN> line 1.
Found  features
Creating the MAGE object
MAGE creation took: 0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)
Writing MAGE-ML
Writing XML took: 0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
Finished

Re: [GeneX-dev] Problem loading cdf file

[Hrishikesh D. <hde...@gm...>]

Lonnnnnnnnngggg Timeeeeeeeeee

Your job is finished. The status is: SUCCESS
Output of the program:

Can't ignore signal CHLD, forcing to default.
Parsing CDF took: 43 wallclock secs (40.81 usr +  0.67 sys = 41.48 CPU)
Found 407536 Reporters, and 407536 Features
Inserting Reporters took: 686 wallclock secs (108.73 usr + 32.71 sys = 
141.44 CPU)
Found 2065 Blank Features
Inserting Features and ArrayDesign took: 1197 wallclock secs (125.51 usr 
+ 57.48 sys = 182.99 CPU)
Finished

Harry Mangalam wrote:
> OK - carefully watching it run on the lappie, typing on another one - 
> this timeimmediately after start, load is staying constant @ ~1.5, 
> but the script is sucking up all the mem (512M) and is now starting 
> to swap like a maniac -.. There goes all the swap, now load is 
> starting to increase to handle all the swapping. OK - now locked up.  
> Final mem usage of the perl script was > 650 MB.  no response left in 
> the keyboard. Had to kill it by powering off the lappie.  thanks be 
> to reiserfs.
>  it looks like to load one of the cdf files you need to have gigs of 
> mem.  I think I'm going to have to try it on my larger system which 
> has 2 GB physical RAM and 4 GB swap.  Letcha know how that goes.
> 
> Hrishi - did you try to load one of the CDF files on one of your 
> systems?  if so did you watch the eventual mem usage?
> 
> On Thursday 09 December 2004 3:07 am, Jason E. Stewart wrote:
> 
>>Harry Mangalam <hj...@ta...> writes:
>>
>>>AFAIK, it wasn't in the loading howto, but I'll check more
>>>carefully.
>>
>>Ah, it was in the HOWTO, but up in the section on loading Array
>>Design's. I've written it so many times I thought I was repeating
>>myself, so I left it out. I think it should be included.
>>
>>
>>>Hmm - so it's loading but it's a 100MB cdf file and it looks like
>>>it has completely socked my lappie.
>>
>>Hmmm.. This is something that seems to happen. Try it again
>>sometime and see if it repeats. I loaded it on genex2 and it
>>loaded. I think it took around 20 minutes
>>
>>
>>>CPU Load is at 20 and it's barely responding anymore - disk light
>>>is on continuously tho - must be swapping like a maniac.  I think
>>>I'll probably have to kill it.
>>
>>I would try it once more. One time I tried to load an ArrayDesign
>>(a GAL file, I think), and it was really taking forever, >~20
>>minutes and GAL files are really short compared to .CDF files. I
>>killed it and restarted it, and it ran in 2 minutes. Dunno why.
>>
>>Cheers,
>>jas.
> 
> 

Re: [GeneX-dev] Problem loading cdf file

[Harry M. <hj...@ta...>]

OK - carefully watching it run on the lappie, typing on another one - 
this timeimmediately after start, load is staying constant @ ~1.5, 
but the script is sucking up all the mem (512M) and is now starting 
to swap like a maniac -.. There goes all the swap, now load is 
starting to increase to handle all the swapping. OK - now locked up.  
Final mem usage of the perl script was > 650 MB.  no response left in 
the keyboard. Had to kill it by powering off the lappie.  thanks be 
to reiserfs.
 it looks like to load one of the cdf files you need to have gigs of 
mem.  I think I'm going to have to try it on my larger system which 
has 2 GB physical RAM and 4 GB swap.  Letcha know how that goes.

Hrishi - did you try to load one of the CDF files on one of your 
systems?  if so did you watch the eventual mem usage?

On Thursday 09 December 2004 3:07 am, Jason E. Stewart wrote:
> Harry Mangalam <hj...@ta...> writes:
> > AFAIK, it wasn't in the loading howto, but I'll check more
> > carefully.
>
> Ah, it was in the HOWTO, but up in the section on loading Array
> Design's. I've written it so many times I thought I was repeating
> myself, so I left it out. I think it should be included.
>
> > Hmm - so it's loading but it's a 100MB cdf file and it looks like
> > it has completely socked my lappie.
>
> Hmmm.. This is something that seems to happen. Try it again
> sometime and see if it repeats. I loaded it on genex2 and it
> loaded. I think it took around 20 minutes
>
> > CPU Load is at 20 and it's barely responding anymore - disk light
> > is on continuously tho - must be swapping like a maniac.  I think
> > I'll probably have to kill it.
>
> I would try it once more. One time I tried to load an ArrayDesign
> (a GAL file, I think), and it was really taking forever, >~20
> minutes and GAL files are really short compared to .CDF files. I
> killed it and restarted it, and it ran in 2 minutes. Dunno why.
>
> Cheers,
> jas.

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hj...@ta... 
            <<plain text preferred>>

[GeneX-dev] How to create AD file for Affymetrix data

[Hrishikesh D. <hde...@gm...>]

Hi Jason,

How do i create an ArrayDesign file for Affymetrix?
What needs to be used .CDF!
I copied .CDF to .txt and this is what i get, does this mean a problem 
or "finished" means there is nothing wrong!

deshmuk@genex2:/usr/local/genex/uploads/genex$ perl 
/usr/local/genex/bin/fix-array.pl --in HG_U95Av2.CDF --design 
HG_U95Av2-AD.txt
Use of uninitialized value in numeric eq (==) at 
/usr/local/genex/bin/fix-array.pl line 230, <IN> line 647487.
Finished

Waiting for your reply.

Thanks,
Hrishi



Jason E. Stewart wrote:
> Harry J Mangalam <hj...@ta...> writes:
> 
> 
>>the CDF file I'm trying to load:
>>
>>507 $ ls -l /usr/local/genex/uploads/
> 
>                                ^^^^^^^
> The upload directories are seperate for each user,
> i.e. /usr/local/genex/uploads/<username>,  so try
> 
>    /usr/local/genex/uploads/genex
> 
> Is this an RTFM, or did I write it incorrectly in the HOWTO?
> 
> Cheers,
> jas.
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now. 
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev

Re: [GeneX-dev] Problem loading cdf file

[<ja...@op...>]

Harry J Mangalam <hj...@ta...> writes:

> the CDF file I'm trying to load:
>
> 507 $ ls -l /usr/local/genex/uploads/
                               ^^^^^^^
The upload directories are seperate for each user,
i.e. /usr/local/genex/uploads/<username>,  so try

   /usr/local/genex/uploads/genex

Is this an RTFM, or did I write it incorrectly in the HOWTO?

Cheers,
jas.

[GeneX-dev] Problem loading cdf file

[Harry J M. <hj...@ta...>]

the CDF file I'm trying to load:

507 $ ls -l /usr/local/genex/uploads/
total 96819
-rw-r--r--  1 root     root     99044850 Dec  8 16:10 HG-U133_Plus_2.cdf

is not showing up in the cdf-load.html page - only the txt and xml files that 
are there are showing up.

I've reloaded the page several times and it is (as shown above) is read - all.

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta... 
            <<plain text preferred>>

[GeneX-dev] Re:Interesting Link!

[Hrishikesh D. <hde...@gm...>]

http://dot.ped.med.umich.edu:2000/ourimage/pub/shared/Affymethods.html
http://dot.ped.med.umich.edu:2000/ourimage/rdk/readaffy.html

Re: [GeneX-dev] Unable to test data loading ...

[<ja...@op...>]

Harry Mangalam <hj...@ta...> writes:

> ... due to a lack of data - Marian's student was unable to get the 
> data to me in a form that I could use.  I'm going to try to get some 
> other data files and use them, but it seems unlikely...

Hey Harry,

If you could just go through the HOWTO and follow the examples given
it would be good to comment on the new rationale and how well the
screens are documented.

Cheers,
jas.

[GeneX-dev] Unable to test data loading ...

[Harry M. <hj...@ta...>]

... due to a lack of data - Marian's student was unable to get the 
data to me in a form that I could use.  I'm going to try to get some 
other data files and use them, but it seems unlikely...
-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hj...@ta... 
            <<plain text preferred>>

[GeneX-dev] clean install svn rev 1718

[Harry J M. <hj...@ta...>]

just an FYI - On a recently updated debian unstable system, I got a clean 
install (one of the very few) with rev  1718.

And a mostly clean data load using the 200-short.txt data set with Jason 
backseat driving from Inja.

Waiting for my wife's affy data to come in and then will try to load that data 
before tomorrow's conf.

Regarding that - Jason says that the landline we used yesterday is still 
available so try  that one first so we don't have to deal with batteries 
dying and other cellphone disasters.
-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - hj...@ta... 
            <<plain text preferred>>

[GeneX-dev] test, please ignore

[<ja...@op...>]

testing that I can once again post to the list, ignore me please

[GeneX-dev] Data Loading Nov-24

[<hde...@gm...>]

Hi Jason,

Here is my attempt to upload data.

Attempt # 1
MBA loading was succesful when i used the test_HG_U95A array from the Array (red field) This shouldn't happen because heat-scok data is from yeast i think.

Attempt # 2
Here is the dataset which i used:

* data file: DB/curated_data/HS_120_1.txt
* AD file: DB/curated_data/YE6100.xml (you have to gunzip the .gz file)
* QT Dim file: DB/curated_data/qtdim-mas5.xml

Insert of Experiment was successful, name given was Nov-24-Exp-Upload

Insert AD output:www.affymetrix.com/ye6100
There were no problems.

Genex Job Status Page

Your job is finished. The status is: SUCCESS
Output of the program:

MAGE creation took:73 wallclock secs (70.42 usr +  0.91 sys = 71.33 CPU)
6191 Reporters to insert
In Bio::Genex::ChangeLog::insert_db: pkey = 
Inserting Reporters took: 9 wallclock secs ( 1.94 usr +  0.29 sys =  2.23 CPU)
6191 FeatureReporterMap's to handle
Handling FR Maps took: 1 wallclock secs ( 0.36 usr +  0.38 sys =  0.74 CPU)
1 ArrayDesigns to insert
Inserting array design: www.affymetrix.com/ye6100
Found 6191 features.
In Bio::Genex::ArrayDesign::insert_db: pkey = 
  In Bio::Genex::ChangeLog::insert_db: pkey = 
  In Bio::Genex::ChangeLog::insert_db: pkey = 
Finished
3357428


GeneX DB ArrayManufacture Edit Page
ArrayManufacture Successfully Added for entry: 3369821:Affy-Ye6100


Insert QT-Dim:
Genex Job Status Page

Your job is finished. The status is: SUCCESS
Output of the program:

NOTICE:  CREATE TABLE / UNIQUE will create implicit index "microarraysuite_5_0_mba_fk_key" for table "microarraysuite_5_0"
Finished
3344930


Insert of MBA:
I used the right parameters in the red fields.

Your job is finished. The status is: ERROR

The error status is: 65280
Output of the program:

Using FeatureExtractionSoftware: MicroArraySuite
Using Experiment: Nov-24-Exp-Upload
Using ArrayDesign: www.affymetrix.com/ye6100
Unknown Feature Extraction Software: MicroArraySuite at /usr/local/genex/bin/mbad-insert.pl line 235.

I have couple of question:

Can we test GeneX by trying to upload dataset of our interest?
If so what are the suggested guidelines?

Thanks,
Hrishi

Re: [GeneX-dev] New Data loader

[<hde...@gm...>]

How do i load a .CEL file?!

Hrishi

> 
> I just added back the ability for the data loader to load QuantArray
> data, so now all the old test data loading works again.
> 
> I have some test GenePix data, so I'm adding a .GAL file loader
> (their version of a .CDF file), and after I can load the Feature
> data from the .GAL file, I'll add a .GPR module for the data loader.
> 
> That will give genex the ability to load QuantArray, .CEL file data,
> and .GPR data. As soon as we get other types that need to be
> supported, we can add those as well.
> 
> Please test out the new loaders - load your favorite .CDF file
> (HU133A for example) and load a bunch of Affy data. If it doesn't
> work I want to know soon so I can fix it. I.e., if you find
> something that doesn't work put the .cdf file and the .cel file 
> in a
> publicly readable directory on genex2 and send me an email.
> 
> Cheers,
> jas.
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real 
> users.Discover which products truly live up to the hype. Start 
> reading now. 
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev
> 

Re: [GeneX-dev] CDF loading

[Hrishikesh D. <hde...@gm...>]

Hi Jason,

Where is the documentation/link which tells us how to upload a .CEL file?

Hrishi


Jason E. Stewart wrote:
> Hey All,
> 
> I forgot to mention. The new GUI for loading CDF files can be located
> off the Workspace tab in the Curator Annotation navbox. You have to
>  be logged in as a curator to access it.
> 
> Also, I've eliminated the oh-so-confusing options from the FE
>  Software section of the QT Dimension loader. From now on out we'll
>  have to write a custom loader module whenever anyone wants to add a
>  new FE SW package.
> 
> Currently only the Affy .CEL format is supported. I'll add back
> QuantArray support in a day or two, and then as soon as I can get
> access to some GenePix data, I'll add that as well.
> 
> Cheers,
> jas.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: InterSystems CACHE
> FREE OODBMS DOWNLOAD - A multidimensional database that combines
> robust object and relational technologies, making it a perfect match
> for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev

Re: [GeneX-dev] Affy CDF loading successful

[Hrishikesh D. <hde...@gm...>]

Hi All,

Here are the results for CDF uploading attempt, in short successful.
Will be loading some more soon.
Thanks,
Hrishi

HG_U74A.CDF
Your job is being run by the system

Current run time is 460 seconds.

Output of the program:

Found 407550 Reporters, and 407550 Features
Parsing CDF took: 43 wallclock secs (41.58 usr +  0.87 sys = 42.45 CPU)
Can't ignore signal CHLD, forcing to default.


MG_U74A.CDF
This page should automatically refresh in about 10 seconds.


Your job is being run by the system

Current run time is 541 seconds.

Output of the program:

Found 407864 Reporters, and 407864 Features
Parsing CDF took: 97 wallclock secs (37.52 usr +  3.60 sys = 41.12 CPU)
Can't ignore signal CHLD, forcing to default.


This page should automatically refresh in about 10 seconds.

Jason E. Stewart wrote:
> Hey All,
> 
> I had a talk last week with Allen Day, the author of the BioPerl
> Affymetrix modules. He indicated that his modules are no longer
> supported as he uses BioConductor for his parsing and analysis.
> 
> I didn't want to require BioPerl for loading Affy data - BioPerl is
> *too* big. So I cut out the core CDF and CEL loading code (very
>  short) and added it to Genex.
> 
> After a bit of testing to figure out why it took so bloody long to
> load the .CEL data into the DB, I'm happy to say that loading of CDF
> and CEL data is successful and fast. I only had one test set to work
> with the Hu6800 chip, so if we could please test it with other CDF
> files and CEL files that would be most helpful.
> 
> Cheers,
> jas.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: InterSystems CACHE
> FREE OODBMS DOWNLOAD - A multidimensional database that combines
> robust object and relational technologies, making it a perfect match
> for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev

[GeneX-dev] New Data loader

[Jason E. S. <ja...@op...>]

I just added back the ability for the data loader to load QuantArray
data, so now all the old test data loading works again.

I have some test GenePix data, so I'm adding a .GAL file loader
 (their version of a .CDF file), and after I can load the Feature
 data from the .GAL file, I'll add a .GPR module for the data loader.

That will give genex the ability to load QuantArray, .CEL file data,
and .GPR data. As soon as we get other types that need to be
supported, we can add those as well.

Please test out the new loaders - load your favorite .CDF file
 (HU133A for example) and load a bunch of Affy data. If it doesn't
 work I want to know soon so I can fix it. I.e., if you find
 something that doesn't work put the .cdf file and the .cel file in a
 publicly readable directory on genex2 and send me an email.

Cheers,
jas.

[GeneX-dev] Affy CDF loading successful

[Jason E. S. <ja...@op...>]

Hey All,

I had a talk last week with Allen Day, the author of the BioPerl
Affymetrix modules. He indicated that his modules are no longer
supported as he uses BioConductor for his parsing and analysis.

I didn't want to require BioPerl for loading Affy data - BioPerl is
*too* big. So I cut out the core CDF and CEL loading code (very
 short) and added it to Genex.

After a bit of testing to figure out why it took so bloody long to
load the .CEL data into the DB, I'm happy to say that loading of CDF
and CEL data is successful and fast. I only had one test set to work
with the Hu6800 chip, so if we could please test it with other CDF
files and CEL files that would be most helpful.

Cheers,
jas.

[GeneX-dev] CDF loading

[Jason E. S. <ja...@op...>]

Hey All,

I forgot to mention. The new GUI for loading CDF files can be located
off the Workspace tab in the Curator Annotation navbox. You have to
 be logged in as a curator to access it.

Also, I've eliminated the oh-so-confusing options from the FE
 Software section of the QT Dimension loader. From now on out we'll
 have to write a custom loader module whenever anyone wants to add a
 new FE SW package.

Currently only the Affy .CEL format is supported. I'll add back
QuantArray support in a day or two, and then as soon as I can get
access to some GenePix data, I'll add that as well.

Cheers,
jas.

[GeneX-dev] new version in SVN

[<ja...@op...>]

Hey All,

The new version is now committed to the SVN repository. In the process
of fixing the *huge* security bug in GroupSecLink, I found a number of
other medium and minor bugs. I wrote a *large* test script that tests
all of the required and optional security features of the DB. It tests
all table *types* and most tables. I don't currently test all tables
because after a fresh install not all tables are populated with data -
this seemed low priority to fix, so I will wait.

There should be very few user-visible changes with the small exception
of a new application available under the Admin tab => 'System
Configuration'. This will list the versions of many of the required
pieces of Genex.

This release has been tested on two machines, but not extensively. It
installs and runs all automated tests, but I didn't have time to run a
really thorough set of manual tests.

Please test the install on genex2. If things seem stable after a few
days we should install it on matrix.

Cheers,
jas.