genex-dev — GeneX developers

[GeneX-dev] Critical Security Bug

[<ja...@op...>]

Hey All,

Last week while documenting how to build components for genex, I was
building an application to associate users with groups and I
discovered to my horror that *any* user could add themselves to *any*
group - this is *very* bad as it completely defeats all row-level
security => if I set a private group for myself, any user can decide
to be part of that group, and read or modify any of my private data.

In short here is the problem: members of a group are stored in one
table GroupSecLink, while the groups themselves are in the GroupSec
table. So no one should have permission to add themselves to a group
unless they are in the read/write group for the GroupSec entry. So the
problem is one table controls the row level security of another table.

It was an issue I dealt with a long time ago, but as the security
system grew more complex, I managed to defeat this, and because I was
not properly testing all security issues in my regression tests - I
missed it. I have fixed the problem, and have added tests to ensure
that no security bugs will escape notice again.

The outcome of this is that many tables in Genex need to be
updated, so please don't attempt to install or update an installation
until I can double-test the new code on genex2.

Cheers,
jas.

[GeneX-dev] Re: wretched excess (regex s)

[<ja...@op...>]

Hey All,

Jennifer and Hrishi had written some off-list emails about data
loading. Since this is the *worst* documented part of the system, I
felt that I should make a public reply so that it can help others.

Jennifer Weller <jw...@gm...> writes:

> When you load an array you are asked for a feature string that I think
> is meant to explain how to link feature information to intensity
> values. 

What is a 'feature_identifier'
======

Genex needs a name that is unique across the Array to locate each
Feature. Probe names are often not unique because replicate
spottings. Some data files sequentially number the output rows, and
that is good enough. Many data files do not - so in the test
QuantArray data provided in DB/curated_data I was forced to
concatenate the Row, Column, ArrayRow, and ArrayCol (as a single
string like 'X.Y.Z.N') to create a single unique value.

Why must the feature_identifier be unique
---

Whenever the data loader (Perl/scripts/mbad-insert.pl) reads a row of
data from the input file I have to know what Feature the row
corresponds to because each row in the MBA Data table has a required
Feature foreign key. I first pull the ArrayDesign from the DB, and get
all the Features, then I read the data file row by row. So for every
data row I have to be able to find what Feature the row is
measuring. In order to do the lookup each Feature on the Array must
have a unique name that I can infer from the row of data just read in
from the data file. Once I succesfully lookup the Feature ID, I then
pull only the data columns that are needed (as described by the QT
Dimension) and insert them into the DB with the correct feature_fk.

Without that unique Feature ID we can't handle replicates properly. If
probe names on the Array are unique (no replicates) then they can be
used as the Feature ID for the ArrayDesign.

The only way this can be determined is for a curator to look at the
data files and look at each column and determine the file
structure. If we there is no unique column, but we can guarantee that
all data files will have exactly the same row ordering (each Feature
appears in the same order for each data file) then that can be made to
work as well.

> This currently seems to be 0 in the test example that works,
> and we don't know how to generalize this. 

Brandon King is probably the only other person on the planet that
understands these values as well as me. He had similar issues loading
GPR data last year. He had a number of communications on the list
about it, and he described in detail what all the values meant - I
included them in the current Data Loading HOWTO under the QT Dim
loading section. This is out of data now because the QTD and the FE SW
are now seperate and independent and these values are part of the FE
SW *not* the QTD. Here is what is in the docs:

  Then there are four parameters that are used during the loading
  process to extract the data from tab-delimited data files:

   1. Feature Identifier String - this string determines what data
      columns are to be combined in order to make the unique feature
      identifier stored in the ArrayDesign. When the data loader
      begins, all the Feature's for that ArrayDesign will be loaded
      from the DB in order to retrieve their primary keys. The feature
      identifier is a unique string used to name each feature of the
      ArrayDesign so that when a row of data is read in from the data
      file, it can be associated with the corresponding Feature from
      the ArrayDesign. For example, if the string is "0.1" then the
      first and second columns will be combined with a period between
      them. Many FeatureExtractionSoftware programs print a unique
      feature identifier as the first column, but in some cases
      multiple columns must be used, for example a program that uses
      zone row, zone column, row, and column to identify each Feature
      could use the string, "0.1.2.3".

The description is (unfortunately for you) clear to me - please let me
know how it is confusing and I will gladly change it. In the test
QuantArray files the value used for the 'Feature Identifier String' is
'1.2.3.4' - meaning the second, third, forth, and fifth columns should
be concatenated together with periods in between to create the
feature_identifier so that it can be looked up in from the Feature
table - this value was taken from the ArrayDesign file.

For the MAS4/5 data, I used the string '0' because the first column
(probe names) were always unique.

My appologies, but the data loader takes these values straight from
the DB and feeds it into a Perl regex - and Perl is like many
programming languages that start counting list elements at 0 (thank
you fortran) - this can be changed with minimal effort if it is too
annoying.

> An earlier version indicated one would input the [1,1,1,1]
> zone-subarray location information that I discussed in the last
> e-mail. For these types of arrays this should be unique but, in
> general, x and y locations must be present and unique no matter what
> type of array you have. 

The Feature identifier is determined by the ArrayDesign file and
stored in the Feature table. Whatever the ArrayDesign sets for the
identifier is what we used. The only current restriction is that the
Data Loader has to be able to read in a row of data from the file we
are busy loading and determine what Feature it is measuring using
only the column values for that row. Some data files will have a
single column that is used as a unique identifier, then that value can
be used by itself. In the example Affy files, the first call is a
unique probe or probeset name which is why the string '0' is used - it
tells the data loader to use the value in the zeroth column.

The ArrayDesign's used for the test data were created by me using one
of the data files as input - therefore I was free to choose whatever
identifier was needed from the data columns.

> The array design would then provide the link from location to probe
> id, and annotation would give probe sequence based on probe id.

Yes, the ArrayDesign gives us the link to the Feature's (locations) and
these give us the link to the Reporter's (probes), and the Reporter's
have a 'sequence' column - or optionaly we can use an external DB
entry (like a GenBank accession number) by using the Database,
DatabaseEntry, and DatabaseEntryLink tables.

It would be useful to load a complete Affy ArrayDesign like the YE6100
and show that this can be done.

> The next part is to give a regular expression for recognizing the
> start of the data. In the test example you have ^0, which leaves us
> confused. We had originally thought that this would be the row number
> where the data starts but regex seems to indicate a pattern is
> requested. 

From the HOWTO:

   2. Data Start Regex #1 - this regex is used to determine when to
      begin reading data rows from the input file. Many
      FeatureExtractionSoftware programs have large headers that need
      to be skipped over before entering data. This regex is matched
      against ever line of the input file, and when it matches, it
      signals the end of the header. This means that the next line
      after the line matched by regex #1 will be read as a column of
      data and stored into the DB.

This value is not for the QuantArray test file, but the Affy YE6100
heat shock example. It is not a zero ^0, but a capital letter O, ^O -
not the easiest thing to distinquish. Here is the first line of the
data file we are parsing:

  ORF	Gene	Heat 120'	"Call:1=P, 0=A"	BKGD

it is a header with no initial comment character. The first word is
ORF - which begins with a capital 'O' - the regex ^O means an line
that begins with a capital O (the '^' means begins with and the 'O' is
just an O). So the very first line of the file is matched - this line
is still skipped, and the very next line is read as a data line to be
stored in the DB.

> If this is true and uniqeness is required, then x location
> and y location (given in two columns) would actually be the only
> guaranteed unique combination. We can't use the probe id because of
> replicates. It doesn't matter if the lowest x and y are in the first
> row, since you would give whatever was in the first row and this
> should not be repeated unless something is wrong with the data.

I hope this should be more clear now that the regex confusion is
straightened out - if not I can explain.

> I am not sure what the data start RegEx#2 is supposed to do, unless
> you had decided to use probe ids as the first data start regex and got
> to a replicate and needed to sort them out.

From the HOWTO:

   3. Data Start Regex #2 - some FeatureExtractionSoftware programs
      have complicated headers which are dificult to match with only a
      single regex. In those rare cases, it is possible to specify
      this optional parameter. In that case the data loader will read
      in lines of the data file until regex #1 matches, then it will
      continue reading lines of the file until regex #2 matches,
      signalling the end of the header section. The next line after
      the line matched by regex #2 will be stored into the DB.

This has nothing to do with replicates - every feature identifier must
uniquely specify the Feature otherwise we cannot use replicates. All
regex #2 does is help to skip over really complicated headers - like
the QuantArray headers. The heat shock example file has almost the
simplest header possible, so this is not needed.

> Then the DataEndRegex is the x and y location at the end of the
> file. We have no idea what ^\s means in the test example.

From the HOWTO:

   4. Data End Regex - after the loader begins storing rows of data
      into the DB, it will attempt to match this regex at every
      line. When it matches true, it signals there is no more data to
      be stored. Usually the regex ^$ signifying a blank line is
      sufficient for most programs. 

X a Y location have nothing to do with this - it simply signals that
we have read in all the data rows in the file and we should exit. Some
files have footers past the data section that we don't want to
accidentally read in as data.

> As I said, we also looked at the insert script, and it looks like it
> depends on the QTD for column locations of intensity, background,
> etc. We were confused by the conversion of column names to
> perl-friendly versions, because we would think that you would skip the
> row that has just headings (unless this is a validation step) and use
> the column definitions that are in the QTD.

That is correct, only the QTD names are used. You were looking at the
QTD insert script (Perl/scripts/qtdim-insert.pl) which converts from
whatever column names the user supplies to SQL and Perl friendly ones
- the name of the data file columns must be used as SQL column names
and Perl method names. 

But when the data file is being read in by the Data Loader
(Perl/scripts/mbad-insert.pl) the column names are pulled from the DB
via the QTD - then the insert SQL is created, and the column values
are inserted into the proper MBA Data table - which is also provided
by the QTD. The Data Loader makes no attempt to parse the column
headers from the input data file - that is why we go through the
trouble of making the QTD, we only need to do it once for each FE
Software instead of specifying it every time we need to load a data
file. 

So to be clear - there are three steps: inserting the FE Software into
the DB - this is what the Feature Identifier and the three regex's are
for; inserting the QTD which defines the data table in the DB, and the
DB column names to be used for which column in the data file; and
finally loading data files. 

> It also appeared that the script looks for the pattern to match the
> start of data in the row before the data starts, which may get
> complicated since there is less constraint on all parts of the file
> not directly involved in the measurements.

Sorry when you say 'the script' I'm not clear which script you mean -
give me a file name or full path and it will be easier. In this case I
think you mean the data loader (Perl/scripts/mbad-insert.pl). You are
correct, the start of data is matching the line *before* data
starts. You are correct that it is quite complicated, which is why I
was forced to make two different data start regex's - because the
QuantArray header is very big. 

Perl regex's are very powerful - but they are tricky. If I wanted to,
I could have said: 'match a row that has 15 data values where each is
separated by a tab character' - and then I could just match the start
of data. In my experience this is more trouble to get correct, so I
decided to match the line immediately before, and then begin matching
data. But, in reality it is a simple matter, and it could be done
either way - it is most important to do it in a way that is both
understandable by the users, and easy for a curator to generate a
suficiently fine-grained regular expression so that only the first
data line is matched and no header lines.

If we decide to go the route of having seperate loading modules for
Affy, GenePix, etc., we won't need the regex's any more. I wanted to
save us the trouble of having to write seperate modules, but I think
now that it was a misguided attempt. By having seperate modules we can
better parse the header information and store that in the DB as well.

> The test example is also somewhat confusing because the datafile
> itself seems to have two sections, one which measured values and one
> with actual intensity values. 

Sorry, which data file? DB/curated_data/HS_120.1.txt or
DB/curated_data/200-short.txt? The HS_120 file works with the FE
Software regex's you listed above - they won't work for the QuantArray
example 200-short.txt.

> We are not clear what distinction is being made. It looks like the
> middle section is a simplified subset of the lower section, which
> has a lot of spot morphology indicators and so on, which have been
> removed in the upper measured values chunk. This may not be
> particularly relevant, but we are seeking enlightenment about what
> to pay attention to.

Yes, I agree, the QuantArray file has a very complex header. That is
why it took me two days to design the Data Loader with enough brains
to handle it. If you really want to understand the file, you will have
to ask Karen - she provided it. I asked her just enough questions to
know that we needed the values in the 'Begin Data' section and not the
values in the 'Begin Measurements' section.

Sorry I can't be more helpful than that.

I hope this email helps - if not please continue to ask!

Cheers,
jas.

Re: [GeneX-dev] Updating matrix => test genex2 first

[<ja...@op...>]

Hrishikesh Deshmukh <hde...@gm...> writes:

>> I want to load QT-dim files but documentation needs some examples
>> for every step for QT-dim. Can you give me input values for say
>> /DB/curated_data/quantarray.xml file, so that i can try loading
>> this,i am a little confused and i think i am using wrong input
>> values for the GUI, so thats the reason i am seeing errors.
>> Waiting for the input values.

Hey Hrishi,

As I indicated before, all the input values for all the QT Dimensions
are loaded at install time. The file that controls this is
Perl/scripts/insert-test-data.pl - if you look their they are
listed:

  $cmd = "$SCRIPTS_DIR/qtdim-insert.pl";
  my @params = ("$OPTIONS{dir}/DB/curated_data/quantarray.xml",
		'--name=QuantArray',
		'--abbrev_name=dqa',
		'--version=3.0',
		q[--feat='"1.2.3.4"'],
		'--data_start_regex1=^Begin\\\\s+Data',
		'--data_start_regex2=^Number',
		'--data_end_regex=^End',
	       );

There is a lot of hairy quoting going on there because it is getting
passed to Perl and then to the command shell. It looks like this when
it gets to the shell command line:

	/usr/local/genex2/bin/qtdim-insert.pl --dbname genex2 --username genex --password yeastecol1 /home/jasons/work/genex-server-svn/DB/curated_data/quantarray.xml --name=QuantArray --abbrev_name=dqa --version=3.0 --feat='"1.2.3.4"' --data_start_regex1=^Begin\\s+Data --data_start_regex2=^Number --data_end_regex=^End

Here is what is actually in the DB:

feature_identifier_string:  "1.2.3.4"
data_start_regex1: ^Begin\s+Data
data_start_regex2: ^Number
data_end_regex: ^End
version: 3.0
name: QuantArray

notice how the feature_identifier string has the double quotes as part
of the value, and how the \s+ is part of the value - these are very
important. \s is the perl regex metacharacter for whitespace so \s+
means any number of whitespace, but at least one. The double quotes
are important around the feature identifier string for a reason I
don't understand - but without them the data loader won't create the
features properly - if it becomes high-priority to fix this, I can
debug it, but at the moment it is just an irritation.

I hope this is more clear,
jas.

>>> Hey Harry, Hrishi, Jennifer, Andrew, et. al.
>>>
>>> I think the new version is fairly stable. Can you give it a day or two
>>> of testing on genex2 and then you can decide to update matrix. It will
>>> require DB changes for the new applications to work properly.
>>>
>>> As I wrote in a private email to Jennifer, this release contains all
>>> the needed schema changes that I am aware of. Only testing will reveal
>>> where it needs more work.
>>>
>>> *Please* do read the new Data Processing Protocol HOWTO, and test out
>>> the examples. It is designed to be an Admin/Curator level document
>>> explaining how to set up protocols to allow users to process their
>>> data. I need feedback on:
>>> * how understandable it is
>>> * how much detail is needed (more/less)
>>> * whether the examples work
>>> * whether the whole system makes sense - i.e. will real people be able
>>>   to use it - both curators and users.
>>>
>>> Cheers,
>>> jas.
>>>
>>>
>>> -------------------------------------------------------
>>> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
>>> Project Admins to receive an Apple iPod Mini FREE for your judgement on
>>> who ports your project to Linux PPC the best. Sponsored by IBM.
>>> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
>>> _______________________________________________
>>> Genex-dev mailing list
>>> Gen...@li...
>>> https://lists.sourceforge.net/lists/listinfo/genex-dev
>> -------------------------------------------------------
>> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
>> Project Admins to receive an Apple iPod Mini FREE for your judgement on
>> who ports your project to Linux PPC the best. Sponsored by IBM.
>> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
>> _______________________________________________
>> Genex-dev mailing list
>> Gen...@li...
>> https://lists.sourceforge.net/lists/listinfo/genex-dev
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev

Re: [GeneX-dev] Updating matrix => test genex2 first

[Hrishikesh D. <hde...@gm...>]

Hi Jason,
Please see the msg below and send me the input values.

Hrishi

Hrishikesh Deshmukh wrote:
> Hi Jason,
> 
> I want to load QT-dim files but documentation needs some examples for 
> every step for QT-dim. Can you give me input values for say 
> /DB/curated_data/quantarray.xml file, so that i can try loading this,i 
> am a little confused and i think i am using wrong input values for the 
> GUI, so thats the reason i am seeing errors.
> 
> Waiting for the input values.
> 
> Hrishi
> 
> 
> Jason E. Stewart wrote:
> 
>> Hey Harry, Hrishi, Jennifer, Andrew, et. al.
>>
>> I think the new version is fairly stable. Can you give it a day or two
>> of testing on genex2 and then you can decide to update matrix. It will
>> require DB changes for the new applications to work properly.
>>
>> As I wrote in a private email to Jennifer, this release contains all
>> the needed schema changes that I am aware of. Only testing will reveal
>> where it needs more work.
>>
>> *Please* do read the new Data Processing Protocol HOWTO, and test out
>> the examples. It is designed to be an Admin/Curator level document
>> explaining how to set up protocols to allow users to process their
>> data. I need feedback on:
>> * how understandable it is
>> * how much detail is needed (more/less)
>> * whether the examples work
>> * whether the whole system makes sense - i.e. will real people be able
>>   to use it - both curators and users.
>>
>> Cheers,
>> jas.
>>
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
>> Project Admins to receive an Apple iPod Mini FREE for your judgement on
>> who ports your project to Linux PPC the best. Sponsored by IBM.
>> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
>> _______________________________________________
>> Genex-dev mailing list
>> Gen...@li...
>> https://lists.sourceforge.net/lists/listinfo/genex-dev
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Genex-dev mailing list
> Gen...@li...
> https://lists.sourceforge.net/lists/listinfo/genex-dev

[GeneX-dev] New docs in SVN

[<ja...@op...>]

Hey All,

I've updated the docs/ directory in SVN - it now has all my current
documentation. I haven't yet updated the menu's in the documentation
section to reflect the new docs so you have to read the HTML files
directly in the browser by typing the URL's.

Cheers,
jas.

[GeneX-dev] How to add a new DB table to Genex

[<ja...@op...>]

Hey All,

Here is a response I gave to one of the GMU genex group on how to add
a table to the DB. It may be useful to others.

Cheers,
jas.

--

> I also have some ODBC and database schema questions.... For instance
> how difficult is to add a couple of new tables that control
> workspace and arraytype management.  

Not too dificult, but there is a process that needs to be
followed. The schema definition is in a set of XML files that live in
the DB/xml/ directory in the genex-server project. At DB install time
those files are converted into SQL and the DB is created. Those same
files are used to create a set of Perl modules in Perl/Bio-Genex/ that
match the DB tables one-to-one. Also, the files are used to create the
edit/insert Mason GUI's that live in Mason/login/annotate/edit/.

So if you want to create a new table:
* decide if the table is a USER table, a CURATOR table, or an ADMIN
  table. USER tables must have security groups - one for determining
  who can read the rows in the table, and one for who can modify the
  rows of the table. CURATOR and ADMIN tables don't use this
  mechanism, the data is assumed public (anyone can read it) but it
  can only be modified by someone with CURATOR (or ADMIN) privelege -
  anyone with the proper privelege can modify any row.
* copy one of the files in DB/xml/ and modify it to be what you
  need. Things to pay attention to are foreign keys - they have a
  <column> tag and a <foreigh_key> tag. The fkey tag has a lot of
  attributes that determine how the Perl API handles things and how
  the SQL is written, e.g. cascading deletes, etc. Use an XML
  validator like nsgmls to ensure that is a valid XML files using the
  table.dtd DTD file.
* Test creating SQL use Perl/scripts/xml2sql.pl - if your table
  inherits from a base table you must include them as well, e.g if you
  are creating a USER table named GENEX_FOO you'll need:

  Perl/scripts/xml2sql DB/xml/GENEX_FOO.xml \
    DB/xml/GENEX_DESCRIBABLE.xml DB/xml/GENEX_IDENTIFIABLE.xml \
    DB/xml/GENEX_SECURABLE.xml 

  Sorry 'bout that but my xml2sql() routine won't follow the
  inheritance tree for you. Lack of tuits.  
* Once the SQL looks proper, you should be able to paste the output of
  xml2sql.pl straight into a pgsql buffer and create the tables -
  WARNING: don't include the pieces for the base tables like
  Identifiable, Describable, and Securable - they already exist and
  you'll get an error if you try.

Those are the required steps, the rest are optional in the short run,
but make life easier in the long run.

* run 'make compile_modules' to make a Bio-Genex Perl module for your
  new tables.
* run 'make install_modules' to make to install your new modules
* run Perl/scripts/run-mason-edit-class.pl to create Mason GUI's for
  your new tables.
* add the new Mason GUI's files to etc/MANIFEST.in - only files here
  are copied during 'make install_files'.

>

Re: [GeneX-dev] Problems loading GPR data

[<ja...@op...>]

Brandon King <ki...@ca...> writes:

> The message seems reasonable, but is there a reason why char12 should
> be invalid? If so, should there be an easy way to add another data
> type? Should genex include more by default?

The predefined types are all listed in
Class::ObjectTemplate::DB::Connect in the ontology_type2sql_type()
method.

They are also list in DB/controlled_vocab/PARAM_DataType.xml.

Adding new types is possible, they need to be changed in both places.

Cheers,
jas.

[GeneX-dev] [ genex-Bugs-1040272 ] run-external-process.mason crash bug

[SourceForge.net <no...@so...>]

Bugs item #1040272, was opened at 2004-10-04 16:13
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=1040272&group_id=16453

Category: Mason GUI
Group: None
Status: Open
Resolution: None
Priority: 7
Submitted By: Brandon King (kingb)
Assigned to: Jason E. Stewart (jason_e_stewart)
Summary: run-external-process.mason crash bug

Initial Comment:
Brandon King <ki...@ca...> writes:
>> Hi All,
>>     Every now and then when I submit a web form like
mba-load.html, I
>>     get the following error... I'm sure what I'm
doing that triggers
>>     it, but it happens fairly often. Any one know
what's going on?
[snip]
>> *code stack:*
/usr/local/share/perl/5.8.4/Bio/Genex/HTMLUtils.pm:750
>> /var/www/genex/mason/comps/run-external-process.mason:57

It is a fork problem with mod_perl. I centralized all
code that needs
to run external processes - using system() or backticks
(`` or
qx//). mod_perl is a pretty complicated thing and when
you fork it you have to do a lot of hanky panky -
closing file descriptors, etc., etc.

One of the things that you have to do is tell the
parent process not
to hang around and wait for it's kids to finish -
normally, Unix
processes do this to avoid getting zombies.

I thought my code was finally working - I copied all
the pieces that
were in the mod_perl HOWTO after google found the
problem for me. I'll dig into this ASAP.

Cheers,
jas.




----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=1040272&group_id=16453

[GeneX-dev] [ genex-Bugs-1040270 ] mba-load.html does not support files with spaces

[SourceForge.net <no...@so...>]

Bugs item #1040270, was opened at 2004-10-04 16:08
Message generated for change (Settings changed) made by kingb
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=1040270&group_id=16453

Category: Mason GUI
Group: None
Status: Open
Resolution: None
Priority: 9
Submitted By: Brandon King (kingb)
Assigned to: Jason E. Stewart (jason_e_stewart)
>Summary: mba-load.html does not support files with spaces

Initial Comment:
Brandon King <ki...@ca...> writes:
>> When I tried to upload an MBA, it was unable to read
the file
>> because the file had spaces in it's name. Whether
it's the fault of
>> mbad-insert.pl or mba-load.html, I'm not sure.

actually it's half-way in between. Currently there is a
Mason component
(Mason/comps/run-external-process.mason) that handles
all fork'ing external processes. At one point I used to
quote all parameters and that worked and then I noticed
that in some cases the
quotes were getting entered into the DB as part of the
values. So I left them out.

It looks like file names need to be quoted...

Cheers,
jas.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=1040270&group_id=16453

[GeneX-dev] [ genex-Bugs-1040270 ] mba-load.html does not support files with space

[SourceForge.net <no...@so...>]

Bugs item #1040270, was opened at 2004-10-04 16:08
Message generated for change (Settings changed) made by kingb
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=1040270&group_id=16453

Category: Mason GUI
Group: None
Status: Open
Resolution: None
Priority: 9
Submitted By: Brandon King (kingb)
Assigned to: Jason E. Stewart (jason_e_stewart)
>Summary: mba-load.html does not support files with space

Initial Comment:
Brandon King <ki...@ca...> writes:
>> When I tried to upload an MBA, it was unable to read
the file
>> because the file had spaces in it's name. Whether
it's the fault of
>> mbad-insert.pl or mba-load.html, I'm not sure.

actually it's half-way in between. Currently there is a
Mason component
(Mason/comps/run-external-process.mason) that handles
all fork'ing external processes. At one point I used to
quote all parameters and that worked and then I noticed
that in some cases the
quotes were getting entered into the DB as part of the
values. So I left them out.

It looks like file names need to be quoted...

Cheers,
jas.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=1040270&group_id=16453

[GeneX-dev] [ genex-Bugs-1040270 ] mbad-insert.pl or mba-load.html does not support files with

[SourceForge.net <no...@so...>]

Bugs item #1040270, was opened at 2004-10-04 16:08
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=1040270&group_id=16453

Category: Mason GUI
Group: None
Status: Open
Resolution: None
Priority: 9
Submitted By: Brandon King (kingb)
Assigned to: Jason E. Stewart (jason_e_stewart)
Summary: mbad-insert.pl or mba-load.html does not support files with 

Initial Comment:
Brandon King <ki...@ca...> writes:
>> When I tried to upload an MBA, it was unable to read
the file
>> because the file had spaces in it's name. Whether
it's the fault of
>> mbad-insert.pl or mba-load.html, I'm not sure.

actually it's half-way in between. Currently there is a
Mason component
(Mason/comps/run-external-process.mason) that handles
all fork'ing external processes. At one point I used to
quote all parameters and that worked and then I noticed
that in some cases the
quotes were getting entered into the DB as part of the
values. So I left them out.

It looks like file names need to be quoted...

Cheers,
jas.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=116453&aid=1040270&group_id=16453

Re: [GeneX-dev] Problems loading GPR data

[Brandon K. <ki...@ca...>]

Jason E. Stewart wrote:

>Brandon King <ki...@ca...> writes:
>
>  
>
>>Brandon King wrote:
>>
>>    
>>
>>>Bio::Genex::Connect:/usr/local/share/perl/5.8.4/Bio/Genex/Connect.pm:569:Bio::Genex::Connect::get_ontology_entry:
>>>Couldn't locate OntologyEntry for category: PARAM_DataType, value:
>>>char12, DBI=[[]]
>>>      
>>>
>>So, I figured out genex doesn't have a char12 datatype, but it does
>>have a char16, so I just changed it in the xml file and it works now.
>>    
>>
>
>Good deal. Was the error message reasonable, or could it be made more
>clear?
>
>Cheers,
>jas.
>
>  
>
The message seems reasonable, but is there a reason why char12 should be 
invalid? If so, should there be an easy way to add another data type? 
Should genex include more by default?

-Brandon

Re: [GeneX-dev] Mystery System Error

[<ja...@op...>]

Brandon King <ki...@ca...> writes:

> Hi All,
>     Every now and then when I submit a web form like mba-load.html, I
>     get the following error... I'm sure what I'm doing that triggers
>     it, but it happens fairly often. Any one know what's going on?

[snip]

> *code stack:*  	/usr/local/share/perl/5.8.4/Bio/Genex/HTMLUtils.pm:750
> /var/www/genex/mason/comps/run-external-process.mason:57

Whaaaa I want to cry - I thought I finally fixed this. 

It is a fork problem with mod_perl. I centralized all code that needs
to run external processes - using system() or backticks (`` or
qx//). mod_perl is a pretty complicated thing and when you fork it you
have to do a lot of hanky panky - closing file descriptors, etc., etc.

One of the things that you have to do is tell the parent process not
to hang around and wait for it's kids to finish - normally, Unix
processes do this to avoid getting zombies.

I thought my code was finally working - I copied all the pieces that
were in the mod_perl HOWTO after google found the problem for me. I'll
dig into this ASAP. 

If you file a priority 7 bug on it for me against
run-external-process.mason I'd be grateful.

Cheers,
jas.

Re: [GeneX-dev] Problems loading GPR data

[<ja...@op...>]

Brandon King <ki...@ca...> writes:

> Brandon King wrote:
>
>> Bio::Genex::Connect:/usr/local/share/perl/5.8.4/Bio/Genex/Connect.pm:569:Bio::Genex::Connect::get_ontology_entry:
>> Couldn't locate OntologyEntry for category: PARAM_DataType, value:
>> char12, DBI=[[]]
>
> So, I figured out genex doesn't have a char12 datatype, but it does
> have a char16, so I just changed it in the xml file and it works now.

Good deal. Was the error message reasonable, or could it be made more
clear?

Cheers,
jas.

Re: [GeneX-dev] mbad-insert.pl or mba-load.html does not support files with spaces

[<ja...@op...>]

Brandon King <ki...@ca...> writes:

> When I tried to upload an MBA, it was unable to read the file
> because the file had spaces in it's name. Whether it's the fault of
> mbad-insert.pl or mba-load.html, I'm not sure.

actually it's half-way in between. Currently there is a Mason
component (Mason/comps/run-external-process.mason) that handles all
fork'ing external processes. At one point I used to quote all
parameters and that worked and then I noticed that in some cases the
quotes were getting entered into the DB as part of the values. So I
left them out. 

It looks like file names need to be quoted...

Cheers,
jas.

Re: [GeneX-dev] mbad-insert.pl or mba-load.html does not support files with spaces

[<ja...@op...>]

I forgot to mention. could you enter this as a priority 9 bug against
mba-load.html so that it doesn't get lost?

Cheers,
jas.

Re: [GeneX-dev] Problems loading GPR data

[Brandon K. <ki...@ca...>]

Brandon King wrote:

> I found that in this case it did insert an ArrayDesign... So, I moved 
> on to loading the qtdim for genepix... I used the old qtdim-genepix.pl 
> program to generate the MAGE-ML file to upload. When I used the QTDim 
> web page for uploading, I got the following error:
>
> Your job is finished. The status is: ERROR
>
> The error status is: -1
>
> Output of the program:
>
> Bio::Genex::Connect:/usr/local/share/perl/5.8.4/Bio/Genex/Connect.pm:569:Bio::Genex::Connect::get_ontology_entry: 
> Couldn't locate OntologyEntry for category: PARAM_DataType, value: 
> char12, DBI=[[]]

So, I figured out genex doesn't have a char12 datatype, but it does have 
a char16, so I just changed it in the xml file and it works now.

>
> -Brandon

[GeneX-dev] Mystery System Error

[Brandon K. <ki...@ca...>]

Hi All,
    Every now and then when I submit a web form like mba-load.html, I 
get the following error... I'm sure what I'm doing that triggers it, but 
it happens fairly often. Any one know what's going on?

-Brandon

*System error*

*error:*  	No child processes at 
/usr/local/share/perl/5.8.4/Bio/Genex/HTMLUtils.pm line 750.
*context:*  	
*...*  	
*746:*  	my $mkdir = $Bio::Genex::Config->{GNUMKDIR};
*747:*  	my $dir = $Bio::Genex::Config->{SYSTMP};
*748:*  	my $command = "umask 000 ; $mkdir -p -m 777 $dir";
*749:*  	system($command);
*750:*  	die $! if $?;
*751:*  	
*752:*  	my ($fh,$filename) = File::Temp::tempfile('gxlogXXXXXX',
*753:*  	DIR => $Bio::Genex::Config->{SYSTMP});
*754:*  	
*...*  	

*code stack:*  	/usr/local/share/perl/5.8.4/Bio/Genex/HTMLUtils.pm:750
/var/www/genex/mason/comps/run-external-process.mason:57
/var/www/genex/mason/login/workspace/mba-load.html:159
/usr/share/perl5/HTML/Mason/Request.pm:753
/var/www/genex/mason/login/workspace/mba-load.html:125
/var/www/genex/mason/login/workspace/autohandler:2
/usr/share/perl5/HTML/Mason/Request.pm:753
/usr/local/genex/mason/obj/login/workspace/autohandler:28
/var/www/genex/mason/login/autohandler:38
/usr/share/perl5/HTML/Mason/Request.pm:753
/var/www/genex/mason/login/autohandler:118
/var/www/genex/mason/autohandler:26
/usr/share/perl5/HTML/Mason/Request.pm:753
/var/www/genex/mason/autohandler:92

raw error 
<http://gecweb.caltech.edu/genex/mason/login/workspace/mba-load.html#raw>



No child processes at /usr/local/share/perl/5.8.4/Bio/Genex/HTMLUtils.pm line 750.


Trace begun at /usr/share/perl5/HTML/Mason/Exceptions.pm line 131
HTML::Mason::Exceptions::rethrow_exception('No child processes at /usr/local/share/perl/5.8.4/Bio/Genex/HTMLUtils.pm line 750.^J') called at /usr/local/share/perl/5.8.4/Bio/Genex/HTMLUtils.pm line 750
Bio::Genex::HTMLUtils::create_job_logfile at /var/www/genex/mason/comps/run-external-process.mason line 57
HTML::Mason::Commands::__ANON__('command', '/usr/local/genex/bin/mbad-insert.pl', 'args', 'HASH(0x9ca4b34)') called at /usr/share/perl5/HTML/Mason/Component.pm line 134
HTML::Mason::Component::run('HTML::Mason::Component::FileBased=HASH(0x9aa4fc0)', 'command', '/usr/local/genex/bin/mbad-insert.pl', 'args', 'HASH(0x9ca4b34)') called at /usr/share/perl5/HTML/Mason/Request.pm line 1074
eval {...}('HTML::Mason::Component::FileBased=HASH(0x9aa4fc0)', 'command', '/usr/local/genex/bin/mbad-insert.pl', 'args', 'HASH(0x9ca4b34)') called at /usr/share/perl5/HTML/Mason/Request.pm line 1068
HTML::Mason::Request::comp(undef, undef, 'command', '/usr/local/genex/bin/mbad-insert.pl', 'args', 'HASH(0x9ca4b34)') called at /var/www/genex/mason/login/workspace/mba-load.html line 159

[GeneX-dev] mbad-insert.pl or mba-load.html does not support files with spaces

[Brandon K. <ki...@ca...>]

Hi Jason,
	When I tried to upload an MBA, it was unable to read the file because the file had spaces in it's name. Whether it's the fault of mbad-insert.pl or mba-load.html, I'm not sure.

Here's the error I receive:

/usr/local/genex/bin/mbad-insert.pl: Couldn't open /usr/local/genex/uploads/genex/Slide for reading, 

The file's name was something like 'Slide A.gpr'.

I'll just rename it for now, but I thought you would like to know.

-Brandon

[GeneX-dev] Problems loading GPR data

[Brandon K. <ki...@ca...>]

First tried creating an ArrayDesign for one GPR file, but it failed for 
some unknown reason (see below).

Your job is finished. The status is: ERROR

The error status is: -1

Output of the program:

FATAL ERROR:
FILE:    /usr/local/genex/uploads/genex/a.xml
LINE:    9
COLUMN:  67
MESSAGE: Expected whitespace
 at /usr/local/share/perl/5.8.4/Bio/MAGE/XML/Reader.pm line 307
Issuing rollback() for database handle being DESTROY'd without explicit disconnect().


It could have been that someone modified the file making it unusable.... 
So, after that didn't work I tried an old gpr experiment and got this 
result:

Your job is finished. The status is: ERROR

The error status is: -1

Output of the program:

MAGE creation took:72 wallclock secs (70.17 usr +  0.45 sys = 70.62 CPU)
8599 Reporters to insert
In Bio::Genex::ChangeLog::insert_db: pkey = 
Inserting Reporters took: 8 wallclock secs ( 1.80 usr +  0.26 sys =  2.06 CPU)
8599 FeatureReporterMap's to handle
Handling FR Maps took: 1 wallclock secs ( 0.49 usr +  0.01 sys =  0.50 CPU)
1 ArrayDesigns to insert
Inserting array design: bawilli2001Mar31
Found 9247 features.
In Bio::Genex::ArrayDesign::insert_db: pkey = 
  In Bio::Genex::ChangeLog::insert_db: pkey = 
  In Bio::Genex::ChangeLog::insert_db: pkey = 
Finished
18232


I found that in this case it did insert an ArrayDesign... So, I moved on 
to loading the qtdim for genepix... I used the old qtdim-genepix.pl 
program to generate the MAGE-ML file to upload. When I used the QTDim 
web page for uploading, I got the following error:

Your job is finished. The status is: ERROR

The error status is: -1

Output of the program:

Bio::Genex::Connect:/usr/local/share/perl/5.8.4/Bio/Genex/Connect.pm:569:Bio::Genex::Connect::get_ontology_entry: Couldn't locate OntologyEntry for category: PARAM_DataType, value: char12, DBI=[[]]
 


Atteched is the qtdim xml file that I used. The regex_start1='Block', 
regex_start2='Column', regex_end='^$'. Let me know if there is any other 
information I could give you that might help explain what's going wrong?

-Brandon

[GeneX-dev] Problems loading affy data

[Brandon K. <ki...@ca...>]

Hi All,
    I can't figure out how to upload an Affy ArrayDesign. First we tried 
using Affy's GDAC Exporter SDK to export a MAGE-ML ArrayDesign. We put 
that in the upload directory, went to web site and clicked run. For 
whatever reason, the session seems to timeout or the status page stops 
being accessible long before the process finishes... When it finally 
does finish, no new ArrayDesign shows up.
    I then tried using the fix-array-cel.pl tab2AD-cel.pl to create an 
array design. I then moved the xml file to the upload directory again, 
and ran the process on the website... Here is the error I got...


Your job is finished. The status is: ERROR

The error status is: -1

Output of the program:

MAGE creation took:315 wallclock secs (309.48 usr +  1.74 sys = 311.22 CPU)
No DesignElement package at /usr/local/genex/bin/array-design-insert.pl line 132.
In Bio::Genex::ChangeLog::insert_db: pkey = 
Inserting Reporters took: 0 wallclock secs ( 0.01 usr +  0.01 sys =  0.02 CPU)
No DesignElement package at /usr/local/genex/bin/array-design-insert.pl line 229.
Handling FR Maps took: 0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)
1 ArrayDesigns to insert
Inserting array design: HG_133A
Found 0 features.
In Bio::Genex::ArrayDesign::insert_db: pkey = 
  In Bio::Genex::ChangeLog::insert_db: pkey = 
  In Bio::Genex::ChangeLog::insert_db: pkey = 
Finished
1026
 


Any suggestions on what I should do to get Affy data loaded?

-Brandon

[GeneX-dev] Caltech Genex 2 Install Notes

[Brandon K. <ki...@ca...>]

Caltech Genex 2.x Install Notes
-----------------------
-'make configure' required Hash::case to be installed, no debian
  package exists. Intalled using cpan.

-'make install' died due to Date::manip not being
  installed... installed libdate-manip-perl debian package. Should
  this have been picked up in 'make configure'?

-'make install' died again giving the following error:

/usr/local/genex/bin/create-genex-db.pl: Couldn't open 
/usr/local/genex/lib/DTD/genexml.dtd for reading at 
/usr/local/share/perl/5.8.4/Bio/Genex/XMLUtils.pm line 2340.
BEGIN failed--compilation aborted at 
/usr/local/share/perl/5.8.4/Bio/Genex/XMLUtils.pm line 2366.
Compilation failed in require at /usr/local/genex/bin/create-genex-db.pl 
line 18.
BEGIN failed--compilation aborted at 
/usr/local/genex/bin/create-genex-db.pl line 18.
Died at Perl/scripts/gendb.pl line 115.
 
  FATAL ERROR
 
I got an error when I ran the DB installer.
 
 
!! System Error:  @ line: 371  (su -c '/usr/bin/perl 
/home/king/proj/genex/genex-server/Perl/scripts/db.pl' postgres)
 
  FATAL ERROR
 
I got an error when I ran the DB installer.
 
make: *** [install] Error 9

-Ran 'su -c '/usr/bin/perl
 /home/king/proj/genex/genex-server/Perl/scripts/db.pl' postgres'
 manually and discovered:

/usr/local/genex/bin/create-genex-db.pl: Couldn't open 
/usr/local/genex/lib/DTD/genexml.dtd for reading at 
/usr/local/share/perl/5.8.4/Bio/Genex/XMLUtils.pm line 2340.

Found that the correct path had a lowercase dtd directory. Made
symbolic link to fix... This fix didn't work as it was being removed
by the install script and run again.

The problem turned out to be that genex-server/Bio/Genex directory was
owned by root and the old Config.pm couldn't be overwritten, but not
crash happend, because it found a Config.pm and just used it. Maybe it
should die if it can't write out a new Config.pm?

-Installation Successful.

Now it's time to try uploading data... We'll e-mail more as things progress.

-Brandon

Re: [GeneX-dev] Wellerlab2 install

[<ja...@op...>]

dc...@gm... writes:

> I did get the latest subversion 1555 to install completely today on
> wellerlab2.  I had to use cpan to install a Hash package.  After that
> the system ran with no errors.  I am currently going through the new
> data load system.

Hey Andrew,

Great! I'm glad the install is working - if you have any comments
criticism about how to get it working *well* please email the list. I
think the system is *much* improved thanks to Harry's new
configuration system, but I'm sure it could be even smoother.

Cheers,
jas.

[GeneX-dev] Wellerlab2 install

[<dc...@gm...>]

Harry,

I did get the latest subversion 1555 to install completely today  on wellerlab2.  I had to use cpan to install a Hash package.  After that the system ran with no errors.  I am currently going through the new data load system.

Andrew

Re: [GeneX-dev] Subversion server

[Harry M. <hj...@ta...>]

Sorry Diane, It's up now.

hjm


On Monday 27 September 2004 10:31 am, Diane Trout wrote:
> Hi,
>
> I think the subversion daemon is currently stopped.
>
> Thanks,
> diane

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hj...@ta... 
            <<plain text preferred>>