mged-mage — General MAGE discussion list

Re: [Mged-mage] [Mged-mage2] MGED Jamboree next week - March 27-30!

[Chris S. <sto...@pc...>]

Great idea.
Chris

On Mar 23, 2007, at 1:18 PM, Helen Parkinson wrote:

> I suggest that we do some round tripping of MAGETAB.
>
> Don Maier wrote:
>> Hi Junmin,
>>
>> As I mentioned, it will be great for us to have a chance to review
>> your work with processing MIAME-compliant documents of various sorts.
>>
>> Your concern about MAGE-TAB compliance ties directly into the agenda
>> item under my name: If we have one and only one parser (accepter) for
>> MAGE-TAB, then every question of compliance is settled once and for
>> everyone.
>>
>> Regards,
>> Don Maier
>>
>>
>> On Mar 23, 2007, at 9:16 AM, Junmin Liu wrote:
>>
>>> Hi, Don and Jamboree attendees,
>>> Since last Oct., for one of the requirement of our in-house data
>>> loading pipeline, we developed an extendable framework MR_Ti and a
>>> set of toolkits to help loading MIAMI-compliant documents in
>>> different formats into our GUS schema. Chris has suggested me to
>>> present this on the Jamboree to get feedback and to see if it can be
>>> useful to the communities.
>>>
>>> Note MAGE-Tab is part of our pipeline, we have started to generate
>>> and load MAGE-Tabs to GUS, we will also like to show public the =20
>>> files
>>> we have and make sure they are exchangable and compliant with
>>> MAGE-Tab spec. 1.0 release.
>>>
>>> Thanks,
>>> ---junmin
>>>
>>>
>>>
>>> On Wed, 21 Mar 2007, Don Maier wrote:
>>>
>>>> Hi friends,
>>>>
>>>> We're looking forward to seeing and meeting with you in next =20
>>>> week in
>>>> sunny California. Current forecasts are for a cloudy Tuesday, but a
>>>> sunny remainder of the week in the upper 60's.
>>>>
>>>> Here's my current list of attendees and agenda. Please let me know
>>>> if you wish to contribute any more bodies or agenda items!
>>>>
>>>> Regards,
>>>> Don Maier
>>>>
>>>>
>>>> MAGE Jamboree: March 27 - 30
>>>> Stanford University: Clark Center, Rm. 362
>>>>
>>>> Attendees:
>>>>
>>>> 1) Cathy Ball (SMD)
>>>> 2) Cate Beauheim (Stanford)
>>>> 3) Alvis Brazma (EBI)
>>>> 4) Steve Chervitz (Affymetrix)
>>>> 5) Junmin Liu (U. Penn)
>>>> 6) Don Maier (SMD)
>>>> 7) Tim Rayner (EBI)
>>>> 8) Ugis Sarkans (EBI)
>>>> 9) Paul Spellman (Lawrence Berkeley National Laboratory)
>>>> 10) Joe White (Dana-Farber/Harvard Cancer Center)
>>>>
>>>>
>>>> Agenda:
>>>>
>>>> 1) Tim =96 Final version of MAGE-TAB
>>>>
>>>> 2) Alvis/Ugis =96 MAGE-TAB object model:
>>>>
>>>> a) Possible move to FUGE-TAB
>>>>
>>>> b) Discussion of basic concepts (no formal proposal yet).
>>>>
>>>> 3) Don =96 The =93one-parser=94 requirement and how to satisfy it =
for
>>>> MAGE-TAB:
>>>>
>>>> a) There is thought to be a strong requirement to have exactly one
>>>> MAGE-TAB parser in one body of code in one language that can be =20
>>>> used
>>>> from either a Java or a Perl programming environment. Satisfying
>>>> this requirement would ensure that MAGE-TAB never destroys =20
>>>> itself on
>>>> the shoals of disagreeing acceptors.
>>>>
>>>> b) Proposal for how to satisfy this requirement.
>>> --------------------------------------------------------------------=20=

>>> -----
>>> Take Surveys. Earn Cash. Influence the Future of IT
>>> Join SourceForge.net's Techsay panel and you'll get the chance to
>>> share your
>>> opinions on IT & business topics through brief surveys-and earn cash
>>> http://www.techsay.com/default.php?=20
>>> page=3Djoin.php&p=3Dsourceforge&CID=3DDEVDEV__________________________=
____=20
>>> _________________
>>> <http://www.techsay.com/default.php?=20
>>> page=3Djoin.php&p=3Dsourceforge&CID=3DDEVDEV__________________________=
____=20
>>> _________________>
>>> Mged-mage mailing list
>>> Mge...@li... <mailto:Mged-=20
>>> ma...@li...>
>>> https://lists.sourceforge.net/lists/listinfo/mged-mage
>>
>> Don Maier
>> Sr. Software Designer/Research
>> Dept. of Biochemistry
>> Stanford University School of Medicine
>>
>>
>> ---------------------------------------------------------------------=20=

>> ---
>>
>> ---------------------------------------------------------------------=20=

>> ----
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to =20
>> share your
>> opinions on IT & business topics through brief surveys-and earn cash
>> http://www.techsay.com/default.php?=20
>> page=3Djoin.php&p=3Dsourceforge&CID=3DDEVDEV
>> ---------------------------------------------------------------------=20=

>> ---
>>
>> _______________________________________________
>> Mged-mage mailing list
>> Mge...@li...
>> https://lists.sourceforge.net/lists/listinfo/mged-mage
>>
>
> --=20
> Helen Parkinson, PhD
> Curation Coordinator
> Microarray Informatics Team,
> EBI
>
> EBI 01223 494672
> Skype: helen.parkinson.ebi
>
>
> ----------------------------------------------------------------------=20=

> ---
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to =20
> share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?=20
> page=3Djoin.php&p=3Dsourceforge&CID=3DDEVDEV
> _______________________________________________
> Mged-mage mailing list
> Mge...@li...
> https://lists.sourceforge.net/lists/listinfo/mged-mage

Re: [Mged-mage] MAGE-TAB officail parser

[Miller, M. D (Rosetta) <Michael_Miller@Rosettabio.com>]

hi don,
=20
as this is the real world, one cannot dictate that developers cannot go
off on their own to do what ever they want.  so my thought is that
realistically we can not require something that is unlikely to be
satisfied.
=20
what we can do is release THE official validating parser which if there
is a question as to whether a set of files that claims to be MAGE_TAB
compliant is or  isn't, it will definitively answer yea or nay.
=20
if someone were to find this official parser wasn't efficient enough in
the context of their application they would not be doing their job to
develop one that did work fast enough for their application.  but having
this official validating parser would be wonderful to make sure the
in-house application stayed compliant.
=20
for instance, in-house, we have our own version of the java MAGEstk
parser which is much more efficient and faster than the one in the
official stk.  (actually i'd like to put it in the official stk code but
it got caught up in some unnecessary imports that prevent releasing it
until those can be removed.)
=20
cheers,
michael
=20



________________________________

	From: Don Maier [mailto:dM...@ge...]=20
	Sent: Friday, March 23, 2007 10:31 AM
	To: Miller, Michael D (Rosetta)
	Cc: Junmin Liu; mged-mage2; MGED-mage
	Subject: Re: [Mged-mage] [Mged-mage2] MGED Jamboree next week -
March 27-30!
=09
=09
	Hi Michael,=20

	Thank you for your enthusiastic support for the "one-parser"
requirement  -- at least, in your initial statement about it.

	Two comments about your comments:

	i)  There likely will be a performance tradeoff.   In my view,
any modest performance cost is utterly trivial compared to the gain in
solving this problem once and for everyone.   Of course, there were
those who invoked the specter of performance to justify continuing
continuing careers in assembly language programming.

	ii)  The architecture of a push parser makes it possible to
perform any set of "custom" actions that are conjoined with the parse.
This includes performing operations on your favorite database and
flagging constructs that an installation prefers not to support.

	There *is* a problem, a really big problem, with having more
than one parser.   Two is too many.   I hope that I can clearly convey
this to the group.

	Regards,
	Don M.


	On Mar 23, 2007, at 9:55 AM, Miller, Michael D (Rosetta) wrote:


		hi junmin,

		that's great work, if i could get a copy of the
presentation when it's
		ready, that would be much appreciated.

		hi don,


				a)     There is thought to be a strong
requirement to have=20

			exactly one=20

				MAGE-TAB parser in one body of code in
one language that=20

			can be used from=20

				either a Java or a Perl programming
environment. =20

			Satisfying this requirement=20

				would ensure that MAGE-TAB never
destroys itself on the=20

			shoals of disagreeing=20

				acceptors.


		i think this is a great requirement, similar to the OMG
compliancy
		section, but i have one caveat with the wording.

		yes, we definitely should have as a goal a definitive
MAGE-TAB parser
		that has a mode that verifies that a set of files is
MAGE-TAB compliant
		(including resolving the references between the files).
And it should
		have a mode that developers could certainly use as the
basis of their
		import implementation.


			exactly one=20

				MAGE-TAB parser in one body of code in
one language that=20

			can be used from=20

				either a Java or a Perl programming
environment. =20


		my caveat is that there is no problem having more than
one, often for
		efficiency reasons for an application there would be a
parser tuned to
		specific database schema and additional application
rules for standard,
		compliant MAGE-TAB files  (for instance, the application
rules might
		restrict MAGE-TAB to require certain columns for the
application that in
		general aren't required).

		i would change the wording to:


			exactly one official validating MAGE-TAB parser
in one body of code in

		one language that=20

			can also be used from=20

				either a Java or a Perl programming
environment by any import

		application.

		note that if it is in one language there will be by
necessity, i think,
		a bit of an impedance using it in a different language.

		cheers,
		michael



			-----Original Message-----
			From: mge...@li...=20
=09
[mailto:mge...@li...] On Behalf=20
			Of Junmin Liu
			Sent: Friday, March 23, 2007 9:17 AM
			To: Don Maier
			Cc: mged-mage2; MGED-mage
			Subject: Re: [Mged-mage2] MGED Jamboree next
week - March 27-30!

			Hi, Don and Jamboree attendees,
			Since last Oct., for one of the requirement of
our in-house=20
			data loading=20
			pipeline, we developed an extendable framework
MR_Ti and a set of=20
			toolkits to help loading MIAMI-compliant
documents in=20
			different formats=20
			into our GUS schema. Chris has suggested me to
present this on the=20
			Jamboree to get feedback and to see if it can be
useful to the=20
			communities.

			Note MAGE-Tab is part of our pipeline, we have
started to=20
			generate and=20
			load MAGE-Tabs to GUS, we will also like to show
public the files we=20
			have and make sure they are exchangable and
compliant with=20
			MAGE-Tab spec.=20
			1.0 release.

			Thanks,
			---junmin



			On Wed, 21 Mar 2007, Don Maier wrote:


				Hi friends,

				We're looking forward to seeing and
meeting with you in=20

			next week in sunny=20

				California.   Current forecasts are for
a cloudy Tuesday,=20

			but a sunny=20

				remainder of the week in the upper 60's.

				Here's my current list of attendees and
agenda.   Please=20

			let me know if you=20

				wish to contribute any more bodies or
agenda items!

				Regards,
				Don Maier


				MAGE Jamboree:  March 27 - 30
				Stanford University:  Clark Center, Rm.
362

				Attendees:

				1)    Cathy Ball (SMD)
				2)    Cate Beauheim (Stanford)
				3)    Alvis Brazma (EBI)
				4)    Steve Chervitz (Affymetrix)
				5)    Junmin Liu (U. Penn)
				6)    Don Maier (SMD)
				7)    Tim Rayner (EBI)
				8)    Ugis Sarkans (EBI)
				9)    Paul Spellman (Lawrence Berkeley
National Laboratory)
				10) Joe White (Dana-Farber/Harvard
Cancer Center)


				Agenda:

				1)    Tim - Final version of MAGE-TAB

				2)    Alvis/Ugis - MAGE-TAB object
model:

				a)     Possible move to FUGE-TAB

				b)    Discussion of basic concepts (no
formal proposal yet).

				3)    Don - The "one-parser" requirement
and how to satisfy=20

			it for MAGE-TAB:


				a)     There is thought to be a strong
requirement to have=20

			exactly one=20

				MAGE-TAB parser in one body of code in
one language that=20

			can be used from=20

				either a Java or a Perl programming
environment. =20

			Satisfying this requirement=20

				would ensure that MAGE-TAB never
destroys itself on the=20

			shoals of disagreeing=20

				acceptors.

				b)    Proposal for how to satisfy this
requirement.




=09
------------------------------------------------------------------------
-
		Take Surveys. Earn Cash. Influence the Future of IT
		Join SourceForge.net's Techsay panel and you'll get the
chance to share your
		opinions on IT & business topics through brief
surveys-and earn cash
=09
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDE
V
		_______________________________________________
		Mged-mage mailing list
		Mge...@li...
		https://lists.sourceforge.net/lists/listinfo/mged-mage

=09
=09

Re: [Mged-mage] [Mged-mage2] MGED Jamboree next week - March 27-30!

[Don M. <dM...@ge...>]

Hi Michael,

Thank you for your enthusiastic support for the "one-parser"  
requirement  -- at least, in your initial statement about it.

Two comments about your comments:

i)  There likely will be a performance tradeoff.   In my view, any  
modest performance cost is utterly trivial compared to the gain in  
solving this problem once and for everyone.   Of course, there were  
those who invoked the specter of performance to justify continuing  
continuing careers in assembly language programming.

ii)  The architecture of a push parser makes it possible to perform  
any set of "custom" actions that are conjoined with the parse.   This  
includes performing operations on your favorite database and flagging  
constructs that an installation prefers not to support.

There *is* a problem, a really big problem, with having more than one  
parser.   Two is too many.   I hope that I can clearly convey this to  
the group.

Regards,
Don M.


On Mar 23, 2007, at 9:55 AM, Miller, Michael D (Rosetta) wrote:

> hi junmin,
>
> that's great work, if i could get a copy of the presentation when it's
> ready, that would be much appreciated.
>
> hi don,
>
>>> a)     There is thought to be a strong requirement to have
>> exactly one
>>> MAGE-TAB parser in one body of code in one language that
>> can be used from
>>> either a Java or a Perl programming environment.
>> Satisfying this requirement
>>> would ensure that MAGE-TAB never destroys itself on the
>> shoals of disagreeing
>>> acceptors.
>
> i think this is a great requirement, similar to the OMG compliancy
> section, but i have one caveat with the wording.
>
> yes, we definitely should have as a goal a definitive MAGE-TAB parser
> that has a mode that verifies that a set of files is MAGE-TAB  
> compliant
> (including resolving the references between the files).  And it should
> have a mode that developers could certainly use as the basis of their
> import implementation.
>
>> exactly one
>>> MAGE-TAB parser in one body of code in one language that
>> can be used from
>>> either a Java or a Perl programming environment.
>
> my caveat is that there is no problem having more than one, often for
> efficiency reasons for an application there would be a parser tuned to
> specific database schema and additional application rules for  
> standard,
> compliant MAGE-TAB files  (for instance, the application rules might
> restrict MAGE-TAB to require certain columns for the application  
> that in
> general aren't required).
>
> i would change the wording to:
>
>> exactly one official validating MAGE-TAB parser in one body of  
>> code in
> one language that
>> can also be used from
>>> either a Java or a Perl programming environment by any import
> application.
>
> note that if it is in one language there will be by necessity, i  
> think,
> a bit of an impedance using it in a different language.
>
> cheers,
> michael
>
>
>> -----Original Message-----
>> From: mge...@li...
>> [mailto:mge...@li...] On Behalf
>> Of Junmin Liu
>> Sent: Friday, March 23, 2007 9:17 AM
>> To: Don Maier
>> Cc: mged-mage2; MGED-mage
>> Subject: Re: [Mged-mage2] MGED Jamboree next week - March 27-30!
>>
>> Hi, Don and Jamboree attendees,
>> Since last Oct., for one of the requirement of our in-house
>> data loading
>> pipeline, we developed an extendable framework MR_Ti and a set of
>> toolkits to help loading MIAMI-compliant documents in
>> different formats
>> into our GUS schema. Chris has suggested me to present this on the
>> Jamboree to get feedback and to see if it can be useful to the
>> communities.
>>
>> Note MAGE-Tab is part of our pipeline, we have started to
>> generate and
>> load MAGE-Tabs to GUS, we will also like to show public the files we
>> have and make sure they are exchangable and compliant with
>> MAGE-Tab spec.
>> 1.0 release.
>>
>> Thanks,
>> ---junmin
>>
>>
>>
>> On Wed, 21 Mar 2007, Don Maier wrote:
>>
>>> Hi friends,
>>>
>>> We're looking forward to seeing and meeting with you in
>> next week in sunny
>>> California.   Current forecasts are for a cloudy Tuesday,
>> but a sunny
>>> remainder of the week in the upper 60's.
>>>
>>> Here's my current list of attendees and agenda.   Please
>> let me know if you
>>> wish to contribute any more bodies or agenda items!
>>>
>>> Regards,
>>> Don Maier
>>>
>>>
>>> MAGE Jamboree:  March 27 - 30
>>> Stanford University:  Clark Center, Rm. 362
>>>
>>> Attendees:
>>>
>>> 1)    Cathy Ball (SMD)
>>> 2)    Cate Beauheim (Stanford)
>>> 3)    Alvis Brazma (EBI)
>>> 4)    Steve Chervitz (Affymetrix)
>>> 5)    Junmin Liu (U. Penn)
>>> 6)    Don Maier (SMD)
>>> 7)    Tim Rayner (EBI)
>>> 8)    Ugis Sarkans (EBI)
>>> 9)    Paul Spellman (Lawrence Berkeley National Laboratory)
>>> 10) Joe White (Dana-Farber/Harvard Cancer Center)
>>>
>>>
>>> Agenda:
>>>
>>> 1)    Tim - Final version of MAGE-TAB
>>>
>>> 2)    Alvis/Ugis - MAGE-TAB object model:
>>>
>>> a)     Possible move to FUGE-TAB
>>>
>>> b)    Discussion of basic concepts (no formal proposal yet).
>>>
>>> 3)    Don - The "one-parser" requirement and how to satisfy
>> it for MAGE-TAB:
>>>
>>> a)     There is thought to be a strong requirement to have
>> exactly one
>>> MAGE-TAB parser in one body of code in one language that
>> can be used from
>>> either a Java or a Perl programming environment.
>> Satisfying this requirement
>>> would ensure that MAGE-TAB never destroys itself on the
>> shoals of disagreeing
>>> acceptors.
>>>
>>> b)    Proposal for how to satisfy this requirement.
>>
>
>
> ---------------------------------------------------------------------- 
> ---
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to  
> share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php? 
> page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Mged-mage mailing list
> Mge...@li...
> https://lists.sourceforge.net/lists/listinfo/mged-mage

Re: [Mged-mage] [Mged-mage2] MGED Jamboree next week - March 27-30!

[Helen P. <par...@eb...>]

I suggest that we do some round tripping of MAGETAB.

Don Maier wrote:
> Hi Junmin,
>
> As I mentioned, it will be great for us to have a chance to review 
> your work with processing MIAME-compliant documents of various sorts.
>
> Your concern about MAGE-TAB compliance ties directly into the agenda 
> item under my name: If we have one and only one parser (accepter) for 
> MAGE-TAB, then every question of compliance is settled once and for 
> everyone.
>
> Regards,
> Don Maier
>
>
> On Mar 23, 2007, at 9:16 AM, Junmin Liu wrote:
>
>> Hi, Don and Jamboree attendees,
>> Since last Oct., for one of the requirement of our in-house data 
>> loading pipeline, we developed an extendable framework MR_Ti and a 
>> set of toolkits to help loading MIAMI-compliant documents in 
>> different formats into our GUS schema. Chris has suggested me to 
>> present this on the Jamboree to get feedback and to see if it can be 
>> useful to the communities.
>>
>> Note MAGE-Tab is part of our pipeline, we have started to generate 
>> and load MAGE-Tabs to GUS, we will also like to show public the files 
>> we have and make sure they are exchangable and compliant with 
>> MAGE-Tab spec. 1.0 release.
>>
>> Thanks,
>> ---junmin
>>
>>
>>
>> On Wed, 21 Mar 2007, Don Maier wrote:
>>
>>> Hi friends,
>>>
>>> We're looking forward to seeing and meeting with you in next week in 
>>> sunny California. Current forecasts are for a cloudy Tuesday, but a 
>>> sunny remainder of the week in the upper 60's.
>>>
>>> Here's my current list of attendees and agenda. Please let me know 
>>> if you wish to contribute any more bodies or agenda items!
>>>
>>> Regards,
>>> Don Maier
>>>
>>>
>>> MAGE Jamboree: March 27 - 30
>>> Stanford University: Clark Center, Rm. 362
>>>
>>> Attendees:
>>>
>>> 1) Cathy Ball (SMD)
>>> 2) Cate Beauheim (Stanford)
>>> 3) Alvis Brazma (EBI)
>>> 4) Steve Chervitz (Affymetrix)
>>> 5) Junmin Liu (U. Penn)
>>> 6) Don Maier (SMD)
>>> 7) Tim Rayner (EBI)
>>> 8) Ugis Sarkans (EBI)
>>> 9) Paul Spellman (Lawrence Berkeley National Laboratory)
>>> 10) Joe White (Dana-Farber/Harvard Cancer Center)
>>>
>>>
>>> Agenda:
>>>
>>> 1) Tim – Final version of MAGE-TAB
>>>
>>> 2) Alvis/Ugis – MAGE-TAB object model:
>>>
>>> a) Possible move to FUGE-TAB
>>>
>>> b) Discussion of basic concepts (no formal proposal yet).
>>>
>>> 3) Don – The “one-parser” requirement and how to satisfy it for 
>>> MAGE-TAB:
>>>
>>> a) There is thought to be a strong requirement to have exactly one 
>>> MAGE-TAB parser in one body of code in one language that can be used 
>>> from either a Java or a Perl programming environment. Satisfying 
>>> this requirement would ensure that MAGE-TAB never destroys itself on 
>>> the shoals of disagreeing acceptors.
>>>
>>> b) Proposal for how to satisfy this requirement.
>> -------------------------------------------------------------------------
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to 
>> share your
>> opinions on IT & business topics through brief surveys-and earn cash
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV_______________________________________________ 
>> <http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV_______________________________________________>
>> Mged-mage mailing list
>> Mge...@li... <mailto:Mge...@li...>
>> https://lists.sourceforge.net/lists/listinfo/mged-mage
>
> Don Maier
> Sr. Software Designer/Research
> Dept. of Biochemistry
> Stanford University School of Medicine
>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> ------------------------------------------------------------------------
>
> _______________________________________________
> Mged-mage mailing list
> Mge...@li...
> https://lists.sourceforge.net/lists/listinfo/mged-mage
>   

-- 
Helen Parkinson, PhD
Curation Coordinator
Microarray Informatics Team, 
EBI

EBI 01223 494672
Skype: helen.parkinson.ebi

Re: [Mged-mage] [Mged-mage2] MGED Jamboree next week - March 27-30!

[Don M. <dM...@ge...>]

Hi Junmin,

As I mentioned, it will be great for us to have a chance to review =20
your work with processing MIAME-compliant documents of various sorts.

Your concern about MAGE-TAB compliance ties directly into the agenda =20
item under my name:   If we have one and only one parser (accepter) =20
for MAGE-TAB, then every question of compliance is settled once and =20
for everyone.

Regards,
Don Maier


On Mar 23, 2007, at 9:16 AM, Junmin Liu wrote:

> Hi, Don and Jamboree attendees,
> Since last Oct., for one of the requirement of our in-house data =20
> loading pipeline, we developed an extendable framework MR_Ti and a =20
> set of toolkits to help loading MIAMI-compliant documents in =20
> different formats into our GUS schema. Chris has suggested me to =20
> present this on the Jamboree to get feedback and to see if it can =20
> be useful to the communities.
>
> Note MAGE-Tab is part of our pipeline, we have started to generate =20
> and load MAGE-Tabs to GUS, we will also like to show public the =20
> files we have and make sure they are exchangable and compliant with =20=

> MAGE-Tab spec. 1.0 release.
>
> Thanks,
> ---junmin
>
>
>
> On Wed, 21 Mar 2007, Don Maier wrote:
>
>> Hi friends,
>>
>> We're looking forward to seeing and meeting with you in next week =20
>> in sunny California.   Current forecasts are for a cloudy Tuesday, =20=

>> but a sunny remainder of the week in the upper 60's.
>>
>> Here's my current list of attendees and agenda.   Please let me =20
>> know if you wish to contribute any more bodies or agenda items!
>>
>> Regards,
>> Don Maier
>>
>>
>> MAGE Jamboree:  March 27 - 30
>> Stanford University:  Clark Center, Rm. 362
>>
>> Attendees:
>>
>> 1)    Cathy Ball (SMD)
>> 2)    Cate Beauheim (Stanford)
>> 3)    Alvis Brazma (EBI)
>> 4)    Steve Chervitz (Affymetrix)
>> 5)    Junmin Liu (U. Penn)
>> 6)    Don Maier (SMD)
>> 7)    Tim Rayner (EBI)
>> 8)    Ugis Sarkans (EBI)
>> 9)    Paul Spellman (Lawrence Berkeley National Laboratory)
>> 10) Joe White (Dana-Farber/Harvard Cancer Center)
>>
>>
>> Agenda:
>>
>> 1)    Tim =96 Final version of MAGE-TAB
>>
>> 2)    Alvis/Ugis =96 MAGE-TAB object model:
>>
>> a)     Possible move to FUGE-TAB
>>
>> b)    Discussion of basic concepts (no formal proposal yet).
>>
>> 3)    Don =96 The =93one-parser=94 requirement and how to satisfy it =
for =20
>> MAGE-TAB:
>>
>> a)     There is thought to be a strong requirement to have exactly =20=

>> one MAGE-TAB parser in one body of code in one language that can =20
>> be used from either a Java or a Perl programming environment.  =20
>> Satisfying this requirement would ensure that MAGE-TAB never =20
>> destroys itself on the shoals of disagreeing acceptors.
>>
>> b)    Proposal for how to satisfy this requirement.
> ----------------------------------------------------------------------=20=

> ---
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to =20
> share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?=20
> page=3Djoin.php&p=3Dsourceforge&CID=3DDEVDEV____________________________=
____=20
> _______________
> Mged-mage mailing list
> Mge...@li...
> https://lists.sourceforge.net/lists/listinfo/mged-mage

Don Maier
Sr. Software Designer/Research
Dept. of Biochemistry
Stanford University School of Medicine

Re: [Mged-mage] [Mged-mage2] MGED Jamboree next week - March 27-30!

[Miller, M. D (Rosetta) <Michael_Miller@Rosettabio.com>]

hi junmin,

that's great work, if i could get a copy of the presentation when it's
ready, that would be much appreciated.

hi don,

> > a)     There is thought to be a strong requirement to have=20
> exactly one=20
> > MAGE-TAB parser in one body of code in one language that=20
> can be used from=20
> > either a Java or a Perl programming environment. =20
> Satisfying this requirement=20
> > would ensure that MAGE-TAB never destroys itself on the=20
> shoals of disagreeing=20
> > acceptors.

i think this is a great requirement, similar to the OMG compliancy
section, but i have one caveat with the wording.

yes, we definitely should have as a goal a definitive MAGE-TAB parser
that has a mode that verifies that a set of files is MAGE-TAB compliant
(including resolving the references between the files).  And it should
have a mode that developers could certainly use as the basis of their
import implementation.

> exactly one=20
> > MAGE-TAB parser in one body of code in one language that=20
> can be used from=20
> > either a Java or a Perl programming environment. =20

my caveat is that there is no problem having more than one, often for
efficiency reasons for an application there would be a parser tuned to
specific database schema and additional application rules for standard,
compliant MAGE-TAB files  (for instance, the application rules might
restrict MAGE-TAB to require certain columns for the application that in
general aren't required).

i would change the wording to:

> exactly one official validating MAGE-TAB parser in one body of code in
one language that=20
> can also be used from=20
> > either a Java or a Perl programming environment by any import
application.

note that if it is in one language there will be by necessity, i think,
a bit of an impedance using it in a different language.

cheers,
michael
=20

> -----Original Message-----
> From: mge...@li...=20
> [mailto:mge...@li...] On Behalf=20
> Of Junmin Liu
> Sent: Friday, March 23, 2007 9:17 AM
> To: Don Maier
> Cc: mged-mage2; MGED-mage
> Subject: Re: [Mged-mage2] MGED Jamboree next week - March 27-30!
>=20
> Hi, Don and Jamboree attendees,
> Since last Oct., for one of the requirement of our in-house=20
> data loading=20
> pipeline, we developed an extendable framework MR_Ti and a set of=20
> toolkits to help loading MIAMI-compliant documents in=20
> different formats=20
> into our GUS schema. Chris has suggested me to present this on the=20
> Jamboree to get feedback and to see if it can be useful to the=20
> communities.
>=20
> Note MAGE-Tab is part of our pipeline, we have started to=20
> generate and=20
> load MAGE-Tabs to GUS, we will also like to show public the files we=20
> have and make sure they are exchangable and compliant with=20
> MAGE-Tab spec.=20
> 1.0 release.
>=20
> Thanks,
> ---junmin
>=20
>=20
>=20
> On Wed, 21 Mar 2007, Don Maier wrote:
>=20
> > Hi friends,
> >
> > We're looking forward to seeing and meeting with you in=20
> next week in sunny=20
> > California.   Current forecasts are for a cloudy Tuesday,=20
> but a sunny=20
> > remainder of the week in the upper 60's.
> >
> > Here's my current list of attendees and agenda.   Please=20
> let me know if you=20
> > wish to contribute any more bodies or agenda items!
> >
> > Regards,
> > Don Maier
> >
> >
> > MAGE Jamboree:  March 27 - 30
> > Stanford University:  Clark Center, Rm. 362
> >
> > Attendees:
> >
> > 1)    Cathy Ball (SMD)
> > 2)    Cate Beauheim (Stanford)
> > 3)    Alvis Brazma (EBI)
> > 4)    Steve Chervitz (Affymetrix)
> > 5)    Junmin Liu (U. Penn)
> > 6)    Don Maier (SMD)
> > 7)    Tim Rayner (EBI)
> > 8)    Ugis Sarkans (EBI)
> > 9)    Paul Spellman (Lawrence Berkeley National Laboratory)
> > 10) Joe White (Dana-Farber/Harvard Cancer Center)
> >
> >
> > Agenda:
> >
> > 1)    Tim - Final version of MAGE-TAB
> >
> > 2)    Alvis/Ugis - MAGE-TAB object model:
> >
> > a)     Possible move to FUGE-TAB
> >
> > b)    Discussion of basic concepts (no formal proposal yet).
> >
> > 3)    Don - The "one-parser" requirement and how to satisfy=20
> it for MAGE-TAB:
> >
> > a)     There is thought to be a strong requirement to have=20
> exactly one=20
> > MAGE-TAB parser in one body of code in one language that=20
> can be used from=20
> > either a Java or a Perl programming environment. =20
> Satisfying this requirement=20
> > would ensure that MAGE-TAB never destroys itself on the=20
> shoals of disagreeing=20
> > acceptors.
> >
> > b)    Proposal for how to satisfy this requirement.
>=20

Re: [Mged-mage] Development release of AE MAGE-TAB parser now available

[Junmin L. <ju...@pc...>]

Hi, Tim,
Upon we release MAGE-Tab 1.0 spec in Stanford, I will make sure the 
mage2tab converter will be upgraded and generate the right MAGE-Tab.

Thanks,
---junmin

On Fri, 23 Mar 2007, Tim Rayner wrote:

> Hi Junmin,
>
> Thanks for the email; just a quick question - is there an updated version of 
> the mage2tab code you would like us to put on the SF page before the 
> jamboree? I think the latest version we have is 0.6.
>
> Cheers,
>
> Tim
>
> On Fri, 23 Mar 2007, Junmin Liu wrote:
>
>> Hi, Tim and all,
>> We, here in CBIL, also developed a suite of toolkit for MAGE-Tab, actually
>> potentially for any kind of MIAMI-complianct mage doc including MAGE-ML,
>> SOFT and MAGE-Tab.
>> 
>> Included in the toolkit are parsers for mage-ml, mage-tab, soft (soon to
>> completed), writers for mage-tab, mage-ml, soft (need to implemented),
>> conversion,visualization and validation tool, GUI interface on top of
>> those tools.
>> 
>> Note we developed a simple object model for consuming the mage-tab, the
>> mage-tab reader will deserialize the mage-tab into the objects. And I
>> understand there are similar agenda items on stanford workshop for these..
>> 
>> The mage2tab we put on tab2mage sourceforge site is part of the toolkit.
>> We already heavily use the toolkit as part of data loading pipeline. We
>> will figure out how to release the toolkit very soon.
>> 
>> ---junmin
>> 
>> On Fri, 23 Mar 2007, Tim Rayner wrote:
>> 
>>> Hi,
>>> 
>>> I'm pleased to be able to announce the initial release of our MAGE-TAB
>>> parser on SourceForge. The parsing scripts, implemented in Perl, are
>>> currently rather alpha-grade (or beta if you're charitable). The code is
>>> available from the following web page:
>>> 
>>> http://sourceforge.net/project/showfiles.php?group_id=120325&package_id=149795&release_id=495838
>>> 
>>> The scripts are included in the Tab2MAGE development package, version
>>> 1.9.7. Installation and usage instructions are included with the download
>>> (see also this link: http://tab2mage.sourceforge.net/#magetab).
>>> 
>>> We have also made available an early release of a script, written by
>>> Faisal Rezwan in our group, which can be used to convert Tab2MAGE
>>> spreadsheets into MAGE-TAB IDF and SDRF documents:
>>> 
>>> http://sourceforge.net/project/showfiles.php?group_id=120325&package_id=226041&release_id=495679
>>> 
>>> Cheers,
>>> 
>>> Tim
>>> 
>>> --
>>> Tim Rayner
>>> ArrayExpress
>>> 
>>> -------------------------------------------------------------------------
>>> Take Surveys. Earn Cash. Influence the Future of IT
>>> Join SourceForge.net's Techsay panel and you'll get the chance to share 
>>> your
>>> opinions on IT & business topics through brief surveys-and earn cash
>>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>>> _______________________________________________
>>> Mged-mage mailing list
>>> Mge...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mged-mage
>>> 
>> 
>> -------------------------------------------------------------------------
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to share 
>> your
>> opinions on IT & business topics through brief surveys-and earn cash
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>> _______________________________________________
>> Mged-mage mailing list
>> Mge...@li...
>> https://lists.sourceforge.net/lists/listinfo/mged-mage
>> 
>

Re: [Mged-mage] [Mged-mage2] MGED Jamboree next week - March 27-30!

[Junmin L. <ju...@pc...>]

Hi, Don and Jamboree attendees,
Since last Oct., for one of the requirement of our in-house data loading=20
pipeline, we developed an extendable framework MR_Ti and a set of=20
toolkits to help loading MIAMI-compliant documents in different formats=20
into our GUS schema. Chris has suggested me to present this on the=20
Jamboree to get feedback and to see if it can be useful to the=20
communities.

Note MAGE-Tab is part of our pipeline, we have started to generate and=20
load MAGE-Tabs to GUS, we will also like to show public the files we=20
have and make sure they are exchangable and compliant with MAGE-Tab spec.=
=20
1.0 release.

Thanks,
---junmin



On Wed, 21 Mar 2007, Don Maier wrote:

> Hi friends,
>
> We're looking forward to seeing and meeting with you in next week in sunn=
y=20
> California.   Current forecasts are for a cloudy Tuesday, but a sunny=20
> remainder of the week in the upper 60's.
>
> Here's my current list of attendees and agenda.   Please let me know if y=
ou=20
> wish to contribute any more bodies or agenda items!
>
> Regards,
> Don Maier
>
>
> MAGE Jamboree:  March 27 - 30
> Stanford University:  Clark Center, Rm. 362
>
> Attendees:
>
> 1)    Cathy Ball (SMD)
> 2)    Cate Beauheim (Stanford)
> 3)    Alvis Brazma (EBI)
> 4)    Steve Chervitz (Affymetrix)
> 5)    Junmin Liu (U. Penn)
> 6)    Don Maier (SMD)
> 7)    Tim Rayner (EBI)
> 8)    Ugis Sarkans (EBI)
> 9)    Paul Spellman (Lawrence Berkeley National Laboratory)
> 10) Joe White (Dana-Farber/Harvard Cancer Center)
>
>
> Agenda:
>
> 1)    Tim =96 Final version of MAGE-TAB
>
> 2)    Alvis/Ugis =96 MAGE-TAB object model:
>
> a)     Possible move to FUGE-TAB
>
> b)    Discussion of basic concepts (no formal proposal yet).
>
> 3)    Don =96 The =93one-parser=94 requirement and how to satisfy it for =
MAGE-TAB:
>
> a)     There is thought to be a strong requirement to have exactly one=20
> MAGE-TAB parser in one body of code in one language that can be used from=
=20
> either a Java or a Perl programming environment.  Satisfying this require=
ment=20
> would ensure that MAGE-TAB never destroys itself on the shoals of disagre=
eing=20
> acceptors.
>
> b)    Proposal for how to satisfy this requirement.

Re: [Mged-mage] Development release of AE MAGE-TAB parser now available

[Tim R. <ra...@eb...>]

Hi Junmin,

Thanks for the email; just a quick question - is there an updated version 
of the mage2tab code you would like us to put on the SF page before the 
jamboree? I think the latest version we have is 0.6.

Cheers,

Tim

On Fri, 23 Mar 2007, Junmin Liu wrote:

> Hi, Tim and all,
> We, here in CBIL, also developed a suite of toolkit for MAGE-Tab, actually
> potentially for any kind of MIAMI-complianct mage doc including MAGE-ML,
> SOFT and MAGE-Tab.
>
> Included in the toolkit are parsers for mage-ml, mage-tab, soft (soon to
> completed), writers for mage-tab, mage-ml, soft (need to implemented),
> conversion,visualization and validation tool, GUI interface on top of
> those tools.
>
> Note we developed a simple object model for consuming the mage-tab, the
> mage-tab reader will deserialize the mage-tab into the objects. And I
> understand there are similar agenda items on stanford workshop for these..
>
> The mage2tab we put on tab2mage sourceforge site is part of the toolkit.
> We already heavily use the toolkit as part of data loading pipeline. We
> will figure out how to release the toolkit very soon.
>
> ---junmin
>
> On Fri, 23 Mar 2007, Tim Rayner wrote:
>
>> Hi,
>>
>> I'm pleased to be able to announce the initial release of our MAGE-TAB
>> parser on SourceForge. The parsing scripts, implemented in Perl, are
>> currently rather alpha-grade (or beta if you're charitable). The code is
>> available from the following web page:
>>
>> http://sourceforge.net/project/showfiles.php?group_id=120325&package_id=149795&release_id=495838
>>
>> The scripts are included in the Tab2MAGE development package, version
>> 1.9.7. Installation and usage instructions are included with the download
>> (see also this link: http://tab2mage.sourceforge.net/#magetab).
>>
>> We have also made available an early release of a script, written by
>> Faisal Rezwan in our group, which can be used to convert Tab2MAGE
>> spreadsheets into MAGE-TAB IDF and SDRF documents:
>>
>> http://sourceforge.net/project/showfiles.php?group_id=120325&package_id=226041&release_id=495679
>>
>> Cheers,
>>
>> Tim
>>
>> --
>> Tim Rayner
>> ArrayExpress
>>
>> -------------------------------------------------------------------------
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to share your
>> opinions on IT & business topics through brief surveys-and earn cash
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>> _______________________________________________
>> Mged-mage mailing list
>> Mge...@li...
>> https://lists.sourceforge.net/lists/listinfo/mged-mage
>>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Mged-mage mailing list
> Mge...@li...
> https://lists.sourceforge.net/lists/listinfo/mged-mage
>

Re: [Mged-mage] Development release of AE MAGE-TAB parser now available

[Junmin L. <ju...@pc...>]

Hi, Tim and all,
We, here in CBIL, also developed a suite of toolkit for MAGE-Tab, actually 
potentially for any kind of MIAMI-complianct mage doc including MAGE-ML, 
SOFT and MAGE-Tab.

Included in the toolkit are parsers for mage-ml, mage-tab, soft (soon to 
completed), writers for mage-tab, mage-ml, soft (need to implemented), 
conversion,visualization and validation tool, GUI interface on top of 
those tools.

Note we developed a simple object model for consuming the mage-tab, the 
mage-tab reader will deserialize the mage-tab into the objects. And I 
understand there are similar agenda items on stanford workshop for these..

The mage2tab we put on tab2mage sourceforge site is part of the toolkit. 
We already heavily use the toolkit as part of data loading pipeline. We 
will figure out how to release the toolkit very soon.

---junmin

On Fri, 23 Mar 2007, Tim Rayner wrote:

> Hi,
>
> I'm pleased to be able to announce the initial release of our MAGE-TAB
> parser on SourceForge. The parsing scripts, implemented in Perl, are
> currently rather alpha-grade (or beta if you're charitable). The code is
> available from the following web page:
>
> http://sourceforge.net/project/showfiles.php?group_id=120325&package_id=149795&release_id=495838
>
> The scripts are included in the Tab2MAGE development package, version
> 1.9.7. Installation and usage instructions are included with the download
> (see also this link: http://tab2mage.sourceforge.net/#magetab).
>
> We have also made available an early release of a script, written by
> Faisal Rezwan in our group, which can be used to convert Tab2MAGE
> spreadsheets into MAGE-TAB IDF and SDRF documents:
>
> http://sourceforge.net/project/showfiles.php?group_id=120325&package_id=226041&release_id=495679
>
> Cheers,
>
> Tim
>
> -- 
> Tim Rayner
> ArrayExpress
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Mged-mage mailing list
> Mge...@li...
> https://lists.sourceforge.net/lists/listinfo/mged-mage
>

[Mged-mage] Development release of AE MAGE-TAB parser now available

[Tim R. <ra...@eb...>]

Hi,

I'm pleased to be able to announce the initial release of our MAGE-TAB 
parser on SourceForge. The parsing scripts, implemented in Perl, are 
currently rather alpha-grade (or beta if you're charitable). The code is 
available from the following web page:

http://sourceforge.net/project/showfiles.php?group_id=120325&package_id=149795&release_id=495838

The scripts are included in the Tab2MAGE development package, version 
1.9.7. Installation and usage instructions are included with the download 
(see also this link: http://tab2mage.sourceforge.net/#magetab).

We have also made available an early release of a script, written by 
Faisal Rezwan in our group, which can be used to convert Tab2MAGE 
spreadsheets into MAGE-TAB IDF and SDRF documents:

http://sourceforge.net/project/showfiles.php?group_id=120325&package_id=226041&release_id=495679

Cheers,

Tim

-- 
Tim Rayner
ArrayExpress

[Mged-mage] MGED Jamboree next week - March 27-30!

[Don M. <dM...@ge...>]

Hi friends,

We're looking forward to seeing and meeting with you in next week in =20
sunny California.   Current forecasts are for a cloudy Tuesday, but a =20=

sunny remainder of the week in the upper 60's.

Here's my current list of attendees and agenda.   Please let me know =20
if you wish to contribute any more bodies or agenda items!

Regards,
Don Maier


MAGE Jamboree:  March 27 - 30
Stanford University:  Clark Center, Rm. 362

Attendees:

1)    Cathy Ball (SMD)
2)    Cate Beauheim (Stanford)
3)    Alvis Brazma (EBI)
4)    Steve Chervitz (Affymetrix)
5)    Junmin Liu (U. Penn)
6)    Don Maier (SMD)
7)    Tim Rayner (EBI)
8)    Ugis Sarkans (EBI)
9)    Paul Spellman (Lawrence Berkeley National Laboratory)
10) Joe White (Dana-Farber/Harvard Cancer Center)


Agenda:

1)    Tim =96 Final version of MAGE-TAB

2)    Alvis/Ugis =96 MAGE-TAB object model:

a)     Possible move to FUGE-TAB

b)    Discussion of basic concepts (no formal proposal yet).

3)    Don =96 The =93one-parser=94 requirement and how to satisfy it for =
=20
MAGE-TAB:

a)     There is thought to be a strong requirement to have exactly =20
one MAGE-TAB parser in one body of code in one language that can be =20
used from either a Java or a Perl programming environment.  =20
Satisfying this requirement would ensure that MAGE-TAB never destroys =20=

itself on the shoals of disagreeing acceptors.

b)    Proposal for how to satisfy this requirement.=

Re: [Mged-mage] [Mged-mage2] MAGE-TAB Illumina question

[Tim R. <ra...@eb...>]

Hi Gavin,

I agree with Michael - this is the way I'd code what you've described in 
MAGE-TAB. The only addition I'd suggest is to add a Label column with the 
label names. A single Labeled Extract can have many Labels; this differs 
from tab2mage, where a separate LE is created for each label. You should 
end up with two rows per hybridization. I've checked, and the current 
parser implementation will support this arrangement (although I don't know 
if the accompanying data file parser copes with idat files as yet). As 
mentioned in my previous email, the feature extraction protocol currently 
has to be combined with the scanning protocol; we may want to consider how 
to fix that at the jamboree.

Cheers,

Tim

-- 

On Sun, 18 Mar 2007, Miller, Michael D (Rosetta) wrote:

> hi gavin,
>
> i'm not that familiar with the genotyping protocols, so this may be off
> base, but i would think one could use the 'Labeled Extract Name' column
> in the order where the labeling occurred.
>
> the tiffs i would imagine would be in the 'Image' column with perhaps a
> Comment column to specify which image it was.
>
> if i understand right, the idat files are the Feature Extracted data
> that's been collapsed per bead type from the raw data, so there could be
> a 'Scan Name' column with two 'Array Data File' columns for the two idat
> files, again with a Comment column for each to differentiate which was
> which.
>
> tim discussed some issues around the feature extraction protocol in his
> 3/16 e-mail which might be relevant.
>
> but i'm not sure what meaning the tiff files would have since the idat
> files are per bead type, not per bead?
>
> cheers,
> michael
>
>> -----Original Message-----
>> From: mge...@li...
>> [mailto:mge...@li...] On Behalf
>> Of Gavin Sherlock
>> Sent: Saturday, March 17, 2007 2:37 PM
>> To: Tim Rayner
>> Cc: mged-mage2; MGED-mage
>> Subject: [Mged-mage2] MAGE-TAB Illumina question
>>
>> Hi Tim,
>>
>> Do you have any example MAGE-TAB files for Illumina Genotyping?
>> Because there is no sample labeling per se, but instead 2 labels are
>> incorporated during an allele specific extension step, after which
>> you get 2 tiffs (one for each channel) and 2 idat files, I was a
>> little unsure how exactly to deal with this in the sdrf file.  Any
>> pointers would be appreciated,
>>
>> Cheers,
>> Gavin
>>
>> --------------------------------------------------------------
>> -----------
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the
>> chance to share your
>> opinions on IT & business topics through brief surveys-and earn cash
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge
>> &CID=DEVDEV
>> _______________________________________________
>> Mged-MAGE2 mailing list
>> Mge...@li...
>> https://lists.sourceforge.net/lists/listinfo/mged-mage2
>>
>>
>
>

Re: [Mged-mage] [Mged-mage2] MAGE-TAB Illumina question

[Miller, M. D (Rosetta) <Michael_Miller@Rosettabio.com>]

hi gavin,

i'm not that familiar with the genotyping protocols, so this may be off
base, but i would think one could use the 'Labeled Extract Name' column
in the order where the labeling occurred.

the tiffs i would imagine would be in the 'Image' column with perhaps a
Comment column to specify which image it was.

if i understand right, the idat files are the Feature Extracted data
that's been collapsed per bead type from the raw data, so there could be
a 'Scan Name' column with two 'Array Data File' columns for the two idat
files, again with a Comment column for each to differentiate which was
which.

tim discussed some issues around the feature extraction protocol in his
3/16 e-mail which might be relevant.

but i'm not sure what meaning the tiff files would have since the idat
files are per bead type, not per bead?

cheers,
michael=20

> -----Original Message-----
> From: mge...@li...=20
> [mailto:mge...@li...] On Behalf=20
> Of Gavin Sherlock
> Sent: Saturday, March 17, 2007 2:37 PM
> To: Tim Rayner
> Cc: mged-mage2; MGED-mage
> Subject: [Mged-mage2] MAGE-TAB Illumina question
>=20
> Hi Tim,
>=20
> Do you have any example MAGE-TAB files for Illumina Genotyping?  =20
> Because there is no sample labeling per se, but instead 2 labels are =20
> incorporated during an allele specific extension step, after which =20
> you get 2 tiffs (one for each channel) and 2 idat files, I was a =20
> little unsure how exactly to deal with this in the sdrf file.  Any =20
> pointers would be appreciated,
>=20
> Cheers,
> Gavin
>=20
> --------------------------------------------------------------
> -----------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the=20
> chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge
> &CID=3DDEVDEV
> _______________________________________________
> Mged-MAGE2 mailing list
> Mge...@li...
> https://lists.sourceforge.net/lists/listinfo/mged-mage2
>=20
>=20

[Mged-mage] MAGE-TAB Illumina question

[Gavin S. <she...@ge...>]

Hi Tim,

Do you have any example MAGE-TAB files for Illumina Genotyping?   
Because there is no sample labeling per se, but instead 2 labels are  
incorporated during an allele specific extension step, after which  
you get 2 tiffs (one for each channel) and 2 idat files, I was a  
little unsure how exactly to deal with this in the sdrf file.  Any  
pointers would be appreciated,

Cheers,
Gavin

Re: [Mged-mage] [Mged-ontologies] ontology annotation properties

[Trish W. <wh...@pc...>]

hi michael,

apols for the delay. for background, what is the need to represent 
annotation properties in MAGE?

> i've been thinking about how ontology annotation properties can be
> handled in MAGE and wanted to run some ideas past you all.
>
> in looking at the w3c owl documentation i found
> (http://www.w3.org/TR/owl-ref/#Annotations):
>
> 	"Five annotation pes are predefined by OWL, namely:
> *	owl:versionInfo
> *	rdfs:label
> *	rdfs:comment
> *	rdfs:seeAlso
> *	rdfs:isDefinedBy"
> plus how addition annotation properties can be defined per ontology.  so
> for any arbitrary ontology, it seems there can be a set of annotation
> properties distinct to that ontology and they can be on classes, and
> object and datatype properties.  so the question becomes how that can be
> encoded in MAGE.
yes, this is true, owl/rdf/rdfs specifies the above set (and more) as well 
as protege itself specifies a few (mainly administrative types), the 
Dublin Core Metadata Initiative (mainly adminsitrative) and also SKOS and 
a newly developed resource to add to the mix, RA/RU. oh, i forgot that the 
conversion of OBO to OWL also uses a self-contained, but not necessarily 
shared set of annotation properties.

as for annotation properties as either object or datatype annotation 
properties this is true of OWL 1.0 (the format that MO exists in and what 
OBI is being developed in). For OWL 1.1, this distinction does not exist 
for annotation property 'types'. I'm not sure when OWL 1.1 will be 
released (I think that it is at least a few months away), but wanted to 
mention this since presumably it will have at least a small impact on 
coding.

> to keep with the current best practice does this seem appropriate?
haven't been following mage so closely lately, but will take a stab at 
this.


>              <OntologyEntry category = "classname" value = "classname">
for clarification, i would say class identifier. for OBI and those 
ontologies converted from OBO to OWL the class identifier will be 
something like OBI_1234, for MO it is the classname itself since that is 
the value of the rdf:ID field, therefore the class identifier.

>                <OntologyReference_assn>
>                  <DatabaseEntry accession="@classname">
>                    <Database_assnref>
>                       <Database_ref
> identifier="www.ontology.com:Database:ontology name" name="ontology
> name"/>
>                    </Database_assnref>
>                  </DatabaseEntry>
>                </OntologyReference_assn>
>                <Associations_assnlist>
>                  <!-- annotation properties for a class -->
>                  <OntologyEntry category = "annotation_properties"
> value = "annotation_properties" >
>                    <Associations_assnlist>
>                      <OntologyEntry category = "label" value = "Class
> Name" />
>                      <OntologyEntry category = "arbitrary property"
> value = "arbitrary value" />
>                    </Associations_assnlist>
>                  </OntologyEntry>
>                  <OntologyEntry category = "datatypepropertyname" value
> = "datatypepropertyname" >
>                    <Associations_assnlist>
>                      <OntologyEntry category = "datatypeproperty" value
> = "value" />
>                      <!-- annotation properties for a datatype property
> -->
>                      <OntologyEntry category = "annotation_properties"
> value = "annotation_properties" >
>                        <Associations_assnlist>
>                          <OntologyEntry category = "label" value =
> "Datatype Property Name" />
>                        </Associations_assnlist>
>                      </OntologyEntry>
>                    </Associations_assnlist>
>                  </OntologyEntry>
>                </Associations_assnlist>
>              </OntologyEntry>

Are you separating out the owl annotation properties from object or 
datatype annotation properties, not sure I follow the comments? As for 
other questions, it prob would be easier to discuss via phone if that is 
ok.

Trish

[Mged-mage] ontology annotation properties

[Miller, M. D (Rosetta) <Michael_Miller@Rosettabio.com>]

hi trish, chris and helen,

i've been thinking about how ontology annotation properties can be
handled in MAGE and wanted to run some ideas past you all.

in looking at the w3c owl documentation i found
(http://www.w3.org/TR/owl-ref/#Annotations):

	"Five annotation properties are predefined by OWL, namely:
*	owl:versionInfo=20
*	rdfs:label=20
*	rdfs:comment=20
*	rdfs:seeAlso=20
*	rdfs:isDefinedBy"=20
plus how addition annotation properties can be defined per ontology.  so
for any arbitrary ontology, it seems there can be a set of annotation
properties distinct to that ontology and they can be on classes, and
object and datatype properties.  so the question becomes how that can be
encoded in MAGE.

to keep with the current best practice does this seem appropriate?

              <OntologyEntry category =3D "classname" value =3D =
"classname">
                <OntologyReference_assn>
                  <DatabaseEntry accession=3D"@classname">
                    <Database_assnref>
                       <Database_ref
identifier=3D"www.ontology.com:Database:ontology name" name=3D"ontology
name"/>
                    </Database_assnref>
                  </DatabaseEntry>
                </OntologyReference_assn>
                <Associations_assnlist>
                  <!-- annotation properties for a class -->
                  <OntologyEntry category =3D "annotation_properties"
value =3D "annotation_properties" >
                    <Associations_assnlist>
                      <OntologyEntry category =3D "label" value =3D =
"Class
Name" />
                      <OntologyEntry category =3D "arbitrary property"
value =3D "arbitrary value" />
                    </Associations_assnlist>
                  </OntologyEntry>
                  <OntologyEntry category =3D "datatypepropertyname" =
value
=3D "datatypepropertyname" >
                    <Associations_assnlist>
                      <OntologyEntry category =3D "datatypeproperty" =
value
=3D "value" />
                      <!-- annotation properties for a datatype property
-->
                      <OntologyEntry category =3D =
"annotation_properties"
value =3D "annotation_properties" >
                        <Associations_assnlist>
                          <OntologyEntry category =3D "label" value =3D
"Datatype Property Name" />
                        </Associations_assnlist>
                      </OntologyEntry>
                    </Associations_assnlist>
                  </OntologyEntry>
                </Associations_assnlist>
              </OntologyEntry>

cheers,
michael

Michael Miller
Lead Software Developer
Rosetta Biosoftware Business Unit
www.rosettabio.com

[Mged-mage] Next MAGE jamboree

[Catherine B. <ba...@ge...>]

Hi all,

In preparation for the upcoming MAGE jamboree, please suggest agenda  
items (Don Maier will keep track of them for us).  Also let him know  
if you are planning to attend.  My current nose count is probably not  
correct, but it is:  Chervitz, Spellman, Li, White, Maier, Beauheim,  
Miller, Rayner (maybe), Parkinson (maybe), Ball.

Details:

> Where:  Clark Center, Room S362, Stanford University,
> Stanford, CA, USA
> When:   9am-5pm, March 27-30, 2007
> Who:    Anyone willing to contribute their time to MGED/MAGE
> projects.  We need to keep attendance down to about 25.
> How much:  No registration required, there is some funding for
> travel .  If you would like funding for travel, contact Cathy Ball
> (ba...@ge...)
> Hotels:  Can be expensive, so do your research.  Stanford has a
> comprehensive list of hotels (http://www.stanford.edu/dept/
> visitorinfo/resources/Lodging.html) and a guest house (http://
> www.stanford.edu/dept/hds/guesthouse//)
> Transportation:  Can be a pain in the neck, but on-campus parking is
> expensive and scarce.  See Stanford's suggestions (http://
> transportation.stanford.edu/)
>

[Mged-mage] OBI Development

[Trish W. <wh...@pc...>]

As most you you know, members of the MGED Ontology working group are
working with other communities on a new project called the Ontology for
Biomedical Investigations (obi.sf.net). This ontology is being developed
to contain both common and domain-specific terms and is being developed
following OBO Foundry best practices (obofoundry.org). One artifact of
this design is that not all term names that a given community would
prefer to see in an annotation application are the term names in the
ontology. As one solution to this issue, the group has proposed an
annotation property to store a user community term name.

We are reviewing the implementation of this annotation property and it is
currently slated to store synonyms, acronyms, abbreviations etc. and some
of these may be further qualified by community/user group. I'm writing to
get feedback from developers as to implementation requests of this
information. Should the various term types be split out into their own
annotation property, should some of these term types not be qualified by a
user community,...?

A listing of these annotation properties can be found at:
https://www.cbil.upenn.edu/obiwiki/index.php/WorkshopMetadata


Trish

[Mged-mage] Fwd: [ANN] CodeSynthesis XSD - open-source XML Schema to C++ compiler

[Jason S. <jas...@gm...>]

Hey All,

Here is an interesting project in light of MAGEstk - it's also nice
that it uses Xerces...

Cheers, jas.

---------- Forwarded message ----------
From: Boris Kolpackov <bo...@co...>
Date: Jan 24, 2007 7:44 PM
Subject: [ANN] CodeSynthesis XSD - open-source XML Schema to C++ compiler
To: c-...@xe...



Hi,

I am pleased to announce the availability of CodeSynthesis XSD 2.3.1.

CodeSynthesis XSD is an open-source, cross-platform W3C XML Schema to
C++ data binding compiler. Provided with a schema, it generates C++
classes that represent the given vocabulary as well as parsing and
serialization code. You can then access the data stored in XML using
types and functions that semantically correspond to your application
domain rather than dealing with elements, attributes, and text in a
direct representation of XML such as DOM or SAX.

XSD supports both in-memory and stream-oriented processing models by
implementing two C++ mappings: C++/Tree and C++/Parser. The C++/Tree
mapping represents the information stored in XML instance documents
as a tree-like, in-memory data structure. The C++/Parser mapping
generates parser templates for data types defined in XML Schema. Using
these parser templates you can build your own in-memory representations
or perform immediate processing of XML instance documents.

XSD is built using Xerces-C++ and both of its C++ mappings use
Xerces-C++ as the underlying XML parser (C++/Parser can also use
Expat).

XSD is available on AIX, GNU/Linux, HP-UX, Mac OS X, Solaris, and Windows.
Supported C++ compilers include: GNU g++, HP aCC, IBM XL C++, Intel C++,
Sun C++, and MS Visual C++.

More information as well as precompiled binaries for all supported
platforms are available from


http://www.codesynthesis.com/products/xsd/



Best Regards,
Boris Kolpackov

Re: [Mged-mage] [Mged-mage2] Collated comments on MAGETAB spec

[Catherine B. <ba...@ge...>]

Hi Tim and others,

I'm getting progressively more worried.  Our goal is not to make MAGE- 
TAB simple "if possible."  It is to make it simple, period.  While it  
might be nice to use MAGE-TAB in an automated fashion to schlep data  
between software applications, the absolutely most important user is  
a bench biologist, end of story.  Please, please, please avoid the  
temptation to start tacking more and more complex data concepts and  
structures on what absolutely needs to be a very simple set of  
spreadsheets that an undergrad or tech can use to annotate a  
reasonably complex set of microarray data in less than an afternoon.

Cathy


On Feb 5, 2007, at 12:44 PM, Junmin Liu wrote:

> Hi, Tim
> I am particularly interested in your following comments.
>
>> I was under the impression that automated pipeline systems would  
>> be using
>> MAGEv2-ML; I'm very concerned that we're mixing up our use cases  
>> here.
>> MAGE-TAB is not supposed to replace MAGEv2-ML, but complement it.  
>> It's a
>> simple tabular format that bench scientists can use (in theory),  
>> *not* a
>> wholesale replacement for the object modelling approach. I worry  
>> greatly
>> that this has been forgotten in the rush to get this specification
>> finished.
>
> I agree with you that MAGE-Tab is not to replace MAGE-ML, but a
> simple way. I think we should stride to make it "SIMPLE" if possible.
>
> In terms of data submission, actually in the past my lab was using
> webforms to annotate the studies and then export MAGE-MLs from DB.  
> Now we
> have MAGE-Tab, my plan is to make the MAGE-Tab as both data loading  
> tool
> and data exporting tool. So the same MAGE-Tab and its associated  
> data can
> be loaded into RAD and also loaded into AE, I think this is feasible.
>
> Also like "MVC", I think MAGE-Tab is like presentation view on the
> underlining model, even we have MAGE-ML v2, or v3, v4, the mapping  
> from
> tab to model are changed, but the MAGE-Tab may not be changed. I  
> could be
> wrong on this.
>
> Thanks,
> ---junmin
>
>
>
>
> ---------------------------------------------------------------------- 
> ---
> Using Tomcat but need to do more? Need to support web services,  
> security?
> Get stuff done quickly with pre-integrated technology to make your  
> job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache  
> Geronimo
> http://sel.as-us.falkag.net/sel? 
> cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Mged-mage mailing list
> Mge...@li...
> https://lists.sourceforge.net/lists/listinfo/mged-mage

Re: [Mged-mage] [Mged-mage2] Collated comments on MAGETAB spec

[Junmin L. <ju...@pc...>]

Hi, Tim
I am particularly interested in your following comments.

> I was under the impression that automated pipeline systems would be using
> MAGEv2-ML; I'm very concerned that we're mixing up our use cases here.
> MAGE-TAB is not supposed to replace MAGEv2-ML, but complement it. It's a
> simple tabular format that bench scientists can use (in theory), *not* a
> wholesale replacement for the object modelling approach. I worry greatly
> that this has been forgotten in the rush to get this specification
> finished.

I agree with you that MAGE-Tab is not to replace MAGE-ML, but a 
simple way. I think we should stride to make it "SIMPLE" if possible.

In terms of data submission, actually in the past my lab was using 
webforms to annotate the studies and then export MAGE-MLs from DB. Now we 
have MAGE-Tab, my plan is to make the MAGE-Tab as both data loading tool 
and data exporting tool. So the same MAGE-Tab and its associated data can 
be loaded into RAD and also loaded into AE, I think this is feasible.

Also like "MVC", I think MAGE-Tab is like presentation view on the 
underlining model, even we have MAGE-ML v2, or v3, v4, the mapping from 
tab to model are changed, but the MAGE-Tab may not be changed. I could be 
wrong on this.

Thanks,
---junmin

Re: [Mged-mage] [Mged-mage2] Collated comments on MAGETAB spec

[Miller, M. D (Rosetta) <Michael_Miller@Rosettabio.com>]

hi tim,

you've addressed my major concerns, and outside of any feedback on the "
issue from others, i'm satisfied

> I should perhaps also comment that I fully expect there to be=20
> a whole slew=20
> of things we've missed in 1.0 which will turn up during=20
> various peoples'=20
> implementations,=20

my main concern was about where the sources came from, so the Provider
column makes me more than happy to wait until a 1.1 to expand on this
and other issues.

cheers,
michael=20

> -----Original Message-----
> From: Tim Rayner [mailto:ra...@eb...]=20
> Sent: Monday, February 05, 2007 6:18 AM
> To: Miller, Michael D (Rosetta)
> Cc: MGED-mage; mged-mage2
> Subject: RE: [Mged-mage2] [Mged-mage] Collated comments on=20
> MAGETAB spec
>=20
>=20
> Hi Michael,
>=20
> Thank you once again for your comments; answers inline below...
>=20
> On Sat, 3 Feb 2007, Miller, Michael D (Rosetta) wrote:
>=20
> > hi tim,
> >
> > thanks for your time.  my initial comments came two weeks=20
> back and my
> > main sticking point, where the Sources were obtained from,=20
> i thought was
> > covered in the specification itself but when i was working=20
> out examples
> > didn't find what i had seen before.  i've taken another=20
> look and isn't
> > this the "PROVIDER" column (section 3.3.6)?  in that case=20
> i'm happy but
> > would like to see the example SDRF file updated to include=20
> this column.
> > to be MIAME compliant, it looks this column must be used=20
> (from the MIAME
> > checklist, caps added by me):
>=20
> Yes, that is indeed the intention. It doesn't provide quite the=20
> fine-grained control of identifiers and namespaces that might be=20
> desirable, of course. Note that I'm not adverse to adding=20
> (optional!) IDF=20
> tags "Investigation Namespace" and "Investigation Authority"=20
> or somesuch=20
> in 1.1, but I do think a wider discussion is needed. I've added a new=20
> Provider column to the example SDRF as requested.
>=20
> I should perhaps also comment that I fully expect there to be=20
> a whole slew=20
> of things we've missed in 1.0 which will turn up during=20
> various peoples'=20
> implementations, and I've been envisioning a 1.1 specification sooner=20
> rather than later (maybe at the Stanford jamboree, or even=20
> sooner?). Now=20
> is definitely the time to be setting the agenda for such an update.
>=20
> > "Samples used, extract preparation and labeling:
> >
> > The origin of each biological sample (e.g., name of the=20
> organism, the
> > PROVIDER of the sample) and its characteristics (e.g., gender, age,
> > developmental stage, strain, or disease state)."
> >
> >> As you wish, although I'm quite sure we can't enforce this
> >> when dealing
> >> with MAGE-TAB documents generated by end users.
> >
> > much appreciated.  if the examples and the tools that=20
> generate templates
> > use this extension, that will help the situation.  in=20
> general, people
> > are used to this, both Affymetrix and GenePix have standard=20
> extensions
> > for their files.
>=20
> Good point - I've also fixed that in the example files.
>=20
> >> I was under the impression that automated pipeline systems
> >> would be using
> >> MAGEv2-ML;
> >
> > you don't give yourself and ArrayExpress enough credit=20
> here.  if it is
> > possible to get a paper published by submitting the data=20
> and annotation
> > using MAGETAB, people are going to expect their GeneExpression
> > applications to take these very same file formats.
> >
> >> I'd rather not have to reinvent the
> >> definition of tab-delimited format; as you've seen, things
> >> get complicated
> >> quite quickly, to the point at which the name of the format
> >> *deceptively*
> >> implies simplicity.
> >
> > my point is that allowing quotes means that MAGETAB is no longer a
> > simple tab-delimited file, it requires a state machine to parse it
> > because " becomes a called out character along with the tab.
>=20
> Yes, to an extent it depends on how "tab-delimited" is=20
> defined, and that=20
> seems to be at the crux of this discussion. If a spreadsheet app user=20
> exports "tab-delimited" text, for better or worse the default is to=20
> sprinkle it liberally with quotes around every text field.
>=20
> >  are tabs
> > allowed between the quotes?  i would think a user would=20
> expect that they
> > are.
>=20
> That would be the intent, yes.
>=20
> > the fact that excel puts quotes around its fields is an artifice of
> > spread-sheet, not tab-delimited format.
> >
> > but as i said i am perfectly willing to go with the consensus here.
>=20
> I whole-heartedly agree (to both points).
>=20
> >> Even so, the MAQC experiment (E-TABM-132 in AE) was quite
> >> successfully
> >> coded on a single Tab2MAGE spreadsheet. I'm not clear why
> >> this is not an
> >> option for other such data sets.
> >
> > the curious thing about this is that BioMaterial 2 and 26 for the
> > primary human samples do have BioSourceProvider attributes provided.
> > also, in principle the experiment isn't MIAME compliant=20
> because the rat
> > samples do not have their PROVIDER listed.
>=20
> I'll look into that, thanks. Most likely the information was=20
> simply not=20
> provided by the submitter. In the meantime the latest (and I=20
> truly hope,=20
> final this time) v1.0 spec is available from the same link as before.
>=20
> Cheers,
>=20
> Tim
>=20
>=20
> > cheers,
> > michael
> >
> >> -----Original Message-----
> >> From: Tim Rayner [mailto:ra...@eb...]
> >> Sent: Thursday, February 01, 2007 3:21 AM
> >> To: Miller, Michael D (Rosetta)
> >> Cc: MGED-mage; mged-mage2
> >> Subject: RE: [Mged-mage2] [Mged-mage] Collated comments on
> >> MAGETAB spec
> >>
> >>
> >> Hi Michael,
> >>
> >> Thanks for your comments. I have attempted to address them below:
> >>
> >>>> 2. Regarding adding a mandatory .idf extension for the IDF
> >>>> file, I think
> >>>> this is not necessary, since it's pretty easy to design a
> >> submission
> >>>> system which would track the IDF without an extension
> >>>
> >>> this is not true of automated pipeline systems, which is a
> >> huge use case
> >>> (as much or more gene expression data and annotation are=20
> loaded via
> >>> these pipelines).  in windows, anyway, it is very easy to
> >> set this up to
> >>> open automatically in excel or whatever spreadsheet program
> >> is desired.
> >>
> >> I was under the impression that automated pipeline systems
> >> would be using
> >> MAGEv2-ML; I'm very concerned that we're mixing up our use
> >> cases here.
> >> MAGE-TAB is not supposed to replace MAGEv2-ML, but complement
> >> it. It's a
> >> simple tabular format that bench scientists can use (in
> >> theory), *not* a
> >> wholesale replacement for the object modelling approach. I
> >> worry greatly
> >> that this has been forgotten in the rush to get this specification
> >> finished.
> >>
> >>> i would be happy with a recommendation that the file
> >> extension be 'idf'.
> >>> if this could be added to the 1st paragraph of section
> >> 3.1.1, that would
> >>> be great.
> >>
> >> As you wish, although I'm quite sure we can't enforce this
> >> when dealing
> >> with MAGE-TAB documents generated by end users.
> >>
> >>>> 3. Use of quotes in IDF and SDRF; while maybe they could be
> >>>> dispensed with
> >>>> in SDRF, they will be needed in IDF to allow e.g. newlines in
> >>>> protocol
> >>>> text (and believe me, users will want this!).
> >>>
> >>> as the addition of information on addition annotation files
> >> in the idf
> >>> was considered a late addition without time for comment, this is a
> >>> rather late addition, it was never part of the=20
> specification proper.
> >>> they can not be made optional, they either must be=20
> mandatory or not.
> >>
> >> I know of at least two well-developed parser modules for perl
> >> which allow
> >> quotes to be optional (Text::CSV_XS and Text::xSV). I have
> >> been assuming
> >> that similar parsers exist for Java and other languages.
> >> Tab2MAGE uses
> >> Text::CSV_XS and has had no problems with optional quoting.
> >>
> >>> i would like to see a handful of fields (like protocol=20
> descriptions)
> >>> where they are required and the rest where they must not be
> >> used but i
> >>> can live with them being mandatory in the IDF and not used
> >> in the SDRF.
> >>
> >> I think we need to use the same treatment for all the files;
> >> not only will
> >> that allow a uniform coding for parsers, but (critically)=20
> it will not
> >> confuse the users as much.
> >>
> >>> there also must be a provision for escaping quotes within=20
> the quoted
> >>> field, believe me, they will definitely occur (the escape=20
> can be as
> >>> simple as '\"' and, less likely, but possible, it will be=20
> the first
> >>> actual character of the field, which is why they must not
> >> be optional
> >>> for fields)
> >>
> >> I agree regarding the \" escape, and I've now added a note on that.
> >> However, a decent parser should be able to distinguish the=20
> following:
> >>
> >> <tab>"your text"<tab>  (" used to quote the field)
> >> <tab>"your" text<tab>  (" used within the content of the field)
> >>
> >> If the second quote in a pair is followed by a tab then that
> >> quote pair is
> >> quoting the field and is not part of the content. This is
> >> pretty much a
> >> solved problem on the perl side of things, although I'd definitely
> >> appreciate feedback if there aren't suitable Java parsers
> >> (this would,
> >> however, surprise me greatly). I'd rather not have to reinvent the
> >> definition of tab-delimited format; as you've seen, things
> >> get complicated
> >> quite quickly, to the point at which the name of the format
> >> *deceptively*
> >> implies simplicity. However, we're never going to be able to get
> >> bench-based end users to comply with a mandatory requirement for or
> >> against quoting here.
> >>
> >> Note that if it turns out there's a consensus against all of
> >> this then I
> >> can remove all the quotes from all the example files and the
> >> discussion in
> >> section 3.1.6. Any code I write will, however, handle quotes
> >> this way,
> >> because I know it works; the alternative does not. I added=20
> it to the
> >> specification because I believe that we can't avoid this=20
> in practice.
> >>
> >>>> 4. We agreed that the authority:namespace component of the
> >>>> resulting MAGE
> >>>> identifiers would be left to the implementation of any
> >>>> parser; while this
> >>>> precludes sharing e.g. Sources defined in MAGE-TAB documents,
> >>>> this is a
> >>>> relatively minor use case. Again, this could be=20
> revisited for a 1.1
> >>>> specification.
> >>>
> >>> this is not a relatively minor use case for many who would
> >> want to use
> >>> MAGETAB--it may be for ArrayExpress right now but it is a=20
> common use
> >>> case for investigation on how to organize a microarray experiment
> >>> (MAQC!!!).  the sharing of sources also occurs for a
> >> variety of other
> >>> reasons.
> >>
> >> I think I should have said "a relatively minor use case for
> >> MAGE-TAB".
> >> Again, what do bench biologists know of authorities and namespaces?
> >> MAGE-TAB is not a replacement for MAGEv2-ML, which I'd have
> >> thought would
> >> be a far more natural tool for large coordinated studies such
> >> as these.
> >> Even so, the MAQC experiment (E-TABM-132 in AE) was quite
> >> successfully
> >> coded on a single Tab2MAGE spreadsheet. I'm not clear why
> >> this is not an
> >> option for other such data sets.
> >>
> >>> this, i thought would be a minor addition to the IDF file
> >> to define the
> >>> default naming authority.  this is not a parser issue--a parser is
> >>> developed in accordance to the specification.  this would
> >> be very bad to
> >>> leave until 1.1, it will cause valuable biological=20
> information to be
> >>> lost.
> >>
> >> We decided very early on that the scope of the identifiers in
> >> the MAGE-TAB
> >> document is limited to within that document. However, if there's a
> >> consensus from everyone that this now needs to be changed,
> >> then of course
> >> we can change it. Given that the deadline for comments was 13
> >> days ago,
> >> though, with all these suggestions made after that deadline,
> >> I'm somewhat
> >> disappointed that we're still having this discussion.
> >>
> >> Best regards,
> >>
> >> Tim
> >>
> >>
> >>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: mge...@li...
> >>>> [mailto:mge...@li...] On Behalf
> >>>> Of Tim Rayner
> >>>> Sent: Wednesday, January 31, 2007 5:46 AM
> >>>> To: 'MGED-mage'; mged-mage2
> >>>> Subject: Re: [Mged-mage2] [Mged-mage] Collated comments on
> >>>> MAGETAB spec
> >>>>
> >>>>
> >>>> Hi,
> >>>>
> >>>> We've discussed the various comments on the previous=20
> version of the
> >>>> MAGE-TAB spec (Jan 8), and it appears that the consensus from
> >>>> ArrayExpress
> >>>> is as follows:
> >>>>
> >>>> 1. Since we need to get a version 1.0 specification
> >> finalised so that
> >>>> implementation deadlines are met, we feel that full support
> >>>> for external
> >>>> annotation files should be deferred to version 1.1. This will
> >>>> allow for
> >>>> more complete discussion of the requirements, in particular
> >>>> in light of
> >>>> any considerations from the FuGE crowd. In the meantime a
> >> minor note
> >>>> has been added to section 3.1.1 regarding suggested use of a
> >>>> Comment[] tag
> >>>> to support these in the meantime.
> >>>>
> >>>> 2. Regarding adding a mandatory .idf extension for the IDF
> >>>> file, I think
> >>>> this is not necessary, since it's pretty easy to design a
> >> submission
> >>>> system which would track the IDF without an extension (e.g.,
> >>>> the current
> >>>> Tab2MAGE submissions system already does this, in effect).
> >>>> Additionally, a
> >>>> new file extension would have to be mapped to Excel or
> >>>> OpenOffice by the
> >>>> end user for it to be any use to them (I believe this is
> >> true of both
> >>>> Windows and Mac). This may not be a huge deal, but it's
> >>>> another barrier to
> >>>> the casual user. Such mapping does not always guarantee that
> >>>> a document
> >>>> opens in the desired application either (OpenOffice, I'm=20
> looking at
> >>>> you...).
> >>>>
> >>>> 3. Use of quotes in IDF and SDRF; while maybe they could be
> >>>> dispensed with
> >>>> in SDRF, they will be needed in IDF to allow e.g. newlines in
> >>>> protocol
> >>>> text (and believe me, users will want this!). The=20
> original Tab2MAGE
> >>>> implementation didn't allow fields to be quoted like this
> >> to preserve
> >>>> special characters (i.e., newlines and tabs), and it was=20
> awful. An
> >>>> additional advantage to using quotes is that "text" date
> >>>> fields such as
> >>>> "2007-01-31" will be preserved by spreadsheet software, while
> >>>> 2007-01-31
> >>>> is often corrupted by such "helpful" applications. I have
> >> added a new
> >>>> section (3.1.6) which briefly discusses use of quotes to
> >> escape data
> >>>> fields. On the bright side, this format tends to be the default
> >>>> for "tab-delimited" export from Excel and OpenOffice in any case.
> >>>>
> >>>> 4. We agreed that the authority:namespace component of the
> >>>> resulting MAGE
> >>>> identifiers would be left to the implementation of any
> >>>> parser; while this
> >>>> precludes sharing e.g. Sources defined in MAGE-TAB documents,
> >>>> this is a
> >>>> relatively minor use case. Again, this could be=20
> revisited for a 1.1
> >>>> specification.
> >>>>
> >>>> 5. Regarding Junmin's comment about array design accessions,
> >>>> it is true
> >>>> that submissions to us will be using ArrayExpress accessions
> >>>> exclusively
> >>>> (at least for the forseeable future) but this is not
> >>>> necessarily true of
> >>>> other users, e.g. those downloading and distributing MAGE-TAB
> >>>> documents
> >>>> from us.
> >>>>
> >>>> 6. As discussed, the specification does indeed allow SDRFs to
> >>>> be split (on
> >>>> any "Name" column) into as many sub-SDRF documents as necessary.
> >>>>
> >>>> I've made the other modifications suggested to the list and
> >>>> put up a new
> >>>> specification document (Jan 31) here:
> >>>>
> >>>>
> >>=20
> http://www.ebi.ac.uk/systems-srv/mp/file-exchange/MAGE-TABv1.0.tar.gz
> >>>>
> >>>> Unless there are serious arguments to the contrary, I
> >>>> personally will be
> >>>> treating this as a finalised version 1.0 specification.
> >>>> Thanks very much
> >>>> for all your comments,
> >>>>
> >>>> Tim
> >>>>
> >>>>
> >>>> --
> >>>> Tim Rayner, Ph.D.
> >>>> Scientific Database Curator
> >>>> Microarray Informatics Team
> >>>> European Bioinformatics Institute
> >>>>
> >>>>
> >>>> --------------------------------------------------------------
> >>>> -----------
> >>>> Take Surveys. Earn Cash. Influence the Future of IT
> >>>> Join SourceForge.net's Techsay panel and you'll get the
> >>>> chance to share your
> >>>> opinions on IT & business topics through brief surveys -
> >> and earn cash
> >>>> =
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge
> >>> &CID=3DDEVDEV
> >>> _______________________________________________
> >>> Mged-MAGE2 mailing list
> >>> Mge...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/mged-mage2
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
>=20
>=20

Re: [Mged-mage] [Mged-mage2] Collated comments on MAGETAB spec

[Tim R. <ra...@eb...>]

Hi Michael,

Thank you once again for your comments; answers inline below...

On Sat, 3 Feb 2007, Miller, Michael D (Rosetta) wrote:

> hi tim,
>
> thanks for your time.  my initial comments came two weeks back and my
> main sticking point, where the Sources were obtained from, i thought was
> covered in the specification itself but when i was working out examples
> didn't find what i had seen before.  i've taken another look and isn't
> this the "PROVIDER" column (section 3.3.6)?  in that case i'm happy but
> would like to see the example SDRF file updated to include this column.
> to be MIAME compliant, it looks this column must be used (from the MIAME
> checklist, caps added by me):

Yes, that is indeed the intention. It doesn't provide quite the 
fine-grained control of identifiers and namespaces that might be 
desirable, of course. Note that I'm not adverse to adding (optional!) IDF 
tags "Investigation Namespace" and "Investigation Authority" or somesuch 
in 1.1, but I do think a wider discussion is needed. I've added a new 
Provider column to the example SDRF as requested.

I should perhaps also comment that I fully expect there to be a whole slew 
of things we've missed in 1.0 which will turn up during various peoples' 
implementations, and I've been envisioning a 1.1 specification sooner 
rather than later (maybe at the Stanford jamboree, or even sooner?). Now 
is definitely the time to be setting the agenda for such an update.

> "Samples used, extract preparation and labeling:
>
> The origin of each biological sample (e.g., name of the organism, the
> PROVIDER of the sample) and its characteristics (e.g., gender, age,
> developmental stage, strain, or disease state)."
>
>> As you wish, although I'm quite sure we can't enforce this
>> when dealing
>> with MAGE-TAB documents generated by end users.
>
> much appreciated.  if the examples and the tools that generate templates
> use this extension, that will help the situation.  in general, people
> are used to this, both Affymetrix and GenePix have standard extensions
> for their files.

Good point - I've also fixed that in the example files.

>> I was under the impression that automated pipeline systems
>> would be using
>> MAGEv2-ML;
>
> you don't give yourself and ArrayExpress enough credit here.  if it is
> possible to get a paper published by submitting the data and annotation
> using MAGETAB, people are going to expect their GeneExpression
> applications to take these very same file formats.
>
>> I'd rather not have to reinvent the
>> definition of tab-delimited format; as you've seen, things
>> get complicated
>> quite quickly, to the point at which the name of the format
>> *deceptively*
>> implies simplicity.
>
> my point is that allowing quotes means that MAGETAB is no longer a
> simple tab-delimited file, it requires a state machine to parse it
> because " becomes a called out character along with the tab.

Yes, to an extent it depends on how "tab-delimited" is defined, and that 
seems to be at the crux of this discussion. If a spreadsheet app user 
exports "tab-delimited" text, for better or worse the default is to 
sprinkle it liberally with quotes around every text field.

>  are tabs
> allowed between the quotes?  i would think a user would expect that they
> are.

That would be the intent, yes.

> the fact that excel puts quotes around its fields is an artifice of
> spread-sheet, not tab-delimited format.
>
> but as i said i am perfectly willing to go with the consensus here.

I whole-heartedly agree (to both points).

>> Even so, the MAQC experiment (E-TABM-132 in AE) was quite
>> successfully
>> coded on a single Tab2MAGE spreadsheet. I'm not clear why
>> this is not an
>> option for other such data sets.
>
> the curious thing about this is that BioMaterial 2 and 26 for the
> primary human samples do have BioSourceProvider attributes provided.
> also, in principle the experiment isn't MIAME compliant because the rat
> samples do not have their PROVIDER listed.

I'll look into that, thanks. Most likely the information was simply not 
provided by the submitter. In the meantime the latest (and I truly hope, 
final this time) v1.0 spec is available from the same link as before.

Cheers,

Tim


> cheers,
> michael
>
>> -----Original Message-----
>> From: Tim Rayner [mailto:ra...@eb...]
>> Sent: Thursday, February 01, 2007 3:21 AM
>> To: Miller, Michael D (Rosetta)
>> Cc: MGED-mage; mged-mage2
>> Subject: RE: [Mged-mage2] [Mged-mage] Collated comments on
>> MAGETAB spec
>>
>>
>> Hi Michael,
>>
>> Thanks for your comments. I have attempted to address them below:
>>
>>>> 2. Regarding adding a mandatory .idf extension for the IDF
>>>> file, I think
>>>> this is not necessary, since it's pretty easy to design a
>> submission
>>>> system which would track the IDF without an extension
>>>
>>> this is not true of automated pipeline systems, which is a
>> huge use case
>>> (as much or more gene expression data and annotation are loaded via
>>> these pipelines).  in windows, anyway, it is very easy to
>> set this up to
>>> open automatically in excel or whatever spreadsheet program
>> is desired.
>>
>> I was under the impression that automated pipeline systems
>> would be using
>> MAGEv2-ML; I'm very concerned that we're mixing up our use
>> cases here.
>> MAGE-TAB is not supposed to replace MAGEv2-ML, but complement
>> it. It's a
>> simple tabular format that bench scientists can use (in
>> theory), *not* a
>> wholesale replacement for the object modelling approach. I
>> worry greatly
>> that this has been forgotten in the rush to get this specification
>> finished.
>>
>>> i would be happy with a recommendation that the file
>> extension be 'idf'.
>>> if this could be added to the 1st paragraph of section
>> 3.1.1, that would
>>> be great.
>>
>> As you wish, although I'm quite sure we can't enforce this
>> when dealing
>> with MAGE-TAB documents generated by end users.
>>
>>>> 3. Use of quotes in IDF and SDRF; while maybe they could be
>>>> dispensed with
>>>> in SDRF, they will be needed in IDF to allow e.g. newlines in
>>>> protocol
>>>> text (and believe me, users will want this!).
>>>
>>> as the addition of information on addition annotation files
>> in the idf
>>> was considered a late addition without time for comment, this is a
>>> rather late addition, it was never part of the specification proper.
>>> they can not be made optional, they either must be mandatory or not.
>>
>> I know of at least two well-developed parser modules for perl
>> which allow
>> quotes to be optional (Text::CSV_XS and Text::xSV). I have
>> been assuming
>> that similar parsers exist for Java and other languages.
>> Tab2MAGE uses
>> Text::CSV_XS and has had no problems with optional quoting.
>>
>>> i would like to see a handful of fields (like protocol descriptions)
>>> where they are required and the rest where they must not be
>> used but i
>>> can live with them being mandatory in the IDF and not used
>> in the SDRF.
>>
>> I think we need to use the same treatment for all the files;
>> not only will
>> that allow a uniform coding for parsers, but (critically) it will not
>> confuse the users as much.
>>
>>> there also must be a provision for escaping quotes within the quoted
>>> field, believe me, they will definitely occur (the escape can be as
>>> simple as '\"' and, less likely, but possible, it will be the first
>>> actual character of the field, which is why they must not
>> be optional
>>> for fields)
>>
>> I agree regarding the \" escape, and I've now added a note on that.
>> However, a decent parser should be able to distinguish the following:
>>
>> <tab>"your text"<tab>  (" used to quote the field)
>> <tab>"your" text<tab>  (" used within the content of the field)
>>
>> If the second quote in a pair is followed by a tab then that
>> quote pair is
>> quoting the field and is not part of the content. This is
>> pretty much a
>> solved problem on the perl side of things, although I'd definitely
>> appreciate feedback if there aren't suitable Java parsers
>> (this would,
>> however, surprise me greatly). I'd rather not have to reinvent the
>> definition of tab-delimited format; as you've seen, things
>> get complicated
>> quite quickly, to the point at which the name of the format
>> *deceptively*
>> implies simplicity. However, we're never going to be able to get
>> bench-based end users to comply with a mandatory requirement for or
>> against quoting here.
>>
>> Note that if it turns out there's a consensus against all of
>> this then I
>> can remove all the quotes from all the example files and the
>> discussion in
>> section 3.1.6. Any code I write will, however, handle quotes
>> this way,
>> because I know it works; the alternative does not. I added it to the
>> specification because I believe that we can't avoid this in practice.
>>
>>>> 4. We agreed that the authority:namespace component of the
>>>> resulting MAGE
>>>> identifiers would be left to the implementation of any
>>>> parser; while this
>>>> precludes sharing e.g. Sources defined in MAGE-TAB documents,
>>>> this is a
>>>> relatively minor use case. Again, this could be revisited for a 1.1
>>>> specification.
>>>
>>> this is not a relatively minor use case for many who would
>> want to use
>>> MAGETAB--it may be for ArrayExpress right now but it is a common use
>>> case for investigation on how to organize a microarray experiment
>>> (MAQC!!!).  the sharing of sources also occurs for a
>> variety of other
>>> reasons.
>>
>> I think I should have said "a relatively minor use case for
>> MAGE-TAB".
>> Again, what do bench biologists know of authorities and namespaces?
>> MAGE-TAB is not a replacement for MAGEv2-ML, which I'd have
>> thought would
>> be a far more natural tool for large coordinated studies such
>> as these.
>> Even so, the MAQC experiment (E-TABM-132 in AE) was quite
>> successfully
>> coded on a single Tab2MAGE spreadsheet. I'm not clear why
>> this is not an
>> option for other such data sets.
>>
>>> this, i thought would be a minor addition to the IDF file
>> to define the
>>> default naming authority.  this is not a parser issue--a parser is
>>> developed in accordance to the specification.  this would
>> be very bad to
>>> leave until 1.1, it will cause valuable biological information to be
>>> lost.
>>
>> We decided very early on that the scope of the identifiers in
>> the MAGE-TAB
>> document is limited to within that document. However, if there's a
>> consensus from everyone that this now needs to be changed,
>> then of course
>> we can change it. Given that the deadline for comments was 13
>> days ago,
>> though, with all these suggestions made after that deadline,
>> I'm somewhat
>> disappointed that we're still having this discussion.
>>
>> Best regards,
>>
>> Tim
>>
>>
>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: mge...@li...
>>>> [mailto:mge...@li...] On Behalf
>>>> Of Tim Rayner
>>>> Sent: Wednesday, January 31, 2007 5:46 AM
>>>> To: 'MGED-mage'; mged-mage2
>>>> Subject: Re: [Mged-mage2] [Mged-mage] Collated comments on
>>>> MAGETAB spec
>>>>
>>>>
>>>> Hi,
>>>>
>>>> We've discussed the various comments on the previous version of the
>>>> MAGE-TAB spec (Jan 8), and it appears that the consensus from
>>>> ArrayExpress
>>>> is as follows:
>>>>
>>>> 1. Since we need to get a version 1.0 specification
>> finalised so that
>>>> implementation deadlines are met, we feel that full support
>>>> for external
>>>> annotation files should be deferred to version 1.1. This will
>>>> allow for
>>>> more complete discussion of the requirements, in particular
>>>> in light of
>>>> any considerations from the FuGE crowd. In the meantime a
>> minor note
>>>> has been added to section 3.1.1 regarding suggested use of a
>>>> Comment[] tag
>>>> to support these in the meantime.
>>>>
>>>> 2. Regarding adding a mandatory .idf extension for the IDF
>>>> file, I think
>>>> this is not necessary, since it's pretty easy to design a
>> submission
>>>> system which would track the IDF without an extension (e.g.,
>>>> the current
>>>> Tab2MAGE submissions system already does this, in effect).
>>>> Additionally, a
>>>> new file extension would have to be mapped to Excel or
>>>> OpenOffice by the
>>>> end user for it to be any use to them (I believe this is
>> true of both
>>>> Windows and Mac). This may not be a huge deal, but it's
>>>> another barrier to
>>>> the casual user. Such mapping does not always guarantee that
>>>> a document
>>>> opens in the desired application either (OpenOffice, I'm looking at
>>>> you...).
>>>>
>>>> 3. Use of quotes in IDF and SDRF; while maybe they could be
>>>> dispensed with
>>>> in SDRF, they will be needed in IDF to allow e.g. newlines in
>>>> protocol
>>>> text (and believe me, users will want this!). The original Tab2MAGE
>>>> implementation didn't allow fields to be quoted like this
>> to preserve
>>>> special characters (i.e., newlines and tabs), and it was awful. An
>>>> additional advantage to using quotes is that "text" date
>>>> fields such as
>>>> "2007-01-31" will be preserved by spreadsheet software, while
>>>> 2007-01-31
>>>> is often corrupted by such "helpful" applications. I have
>> added a new
>>>> section (3.1.6) which briefly discusses use of quotes to
>> escape data
>>>> fields. On the bright side, this format tends to be the default
>>>> for "tab-delimited" export from Excel and OpenOffice in any case.
>>>>
>>>> 4. We agreed that the authority:namespace component of the
>>>> resulting MAGE
>>>> identifiers would be left to the implementation of any
>>>> parser; while this
>>>> precludes sharing e.g. Sources defined in MAGE-TAB documents,
>>>> this is a
>>>> relatively minor use case. Again, this could be revisited for a 1.1
>>>> specification.
>>>>
>>>> 5. Regarding Junmin's comment about array design accessions,
>>>> it is true
>>>> that submissions to us will be using ArrayExpress accessions
>>>> exclusively
>>>> (at least for the forseeable future) but this is not
>>>> necessarily true of
>>>> other users, e.g. those downloading and distributing MAGE-TAB
>>>> documents
>>>> from us.
>>>>
>>>> 6. As discussed, the specification does indeed allow SDRFs to
>>>> be split (on
>>>> any "Name" column) into as many sub-SDRF documents as necessary.
>>>>
>>>> I've made the other modifications suggested to the list and
>>>> put up a new
>>>> specification document (Jan 31) here:
>>>>
>>>>
>> http://www.ebi.ac.uk/systems-srv/mp/file-exchange/MAGE-TABv1.0.tar.gz
>>>>
>>>> Unless there are serious arguments to the contrary, I
>>>> personally will be
>>>> treating this as a finalised version 1.0 specification.
>>>> Thanks very much
>>>> for all your comments,
>>>>
>>>> Tim
>>>>
>>>>
>>>> --
>>>> Tim Rayner, Ph.D.
>>>> Scientific Database Curator
>>>> Microarray Informatics Team
>>>> European Bioinformatics Institute
>>>>
>>>>
>>>> --------------------------------------------------------------
>>>> -----------
>>>> Take Surveys. Earn Cash. Influence the Future of IT
>>>> Join SourceForge.net's Techsay panel and you'll get the
>>>> chance to share your
>>>> opinions on IT & business topics through brief surveys -
>> and earn cash
>>>> http://www.techsay.com/default.php?page=join.php&p=sourceforge
>>> &CID=DEVDEV
>>> _______________________________________________
>>> Mged-MAGE2 mailing list
>>> Mge...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mged-mage2
>>>
>>>
>>>
>>
>>
>
>

Re: [Mged-mage] [Mged-mage2] Collated comments on MAGETAB spec

[Miller, M. D (Rosetta) <Michael_Miller@Rosettabio.com>]

hi tim,

thanks for your time.  my initial comments came two weeks back and my
main sticking point, where the Sources were obtained from, i thought was
covered in the specification itself but when i was working out examples
didn't find what i had seen before.  i've taken another look and isn't
this the "PROVIDER" column (section 3.3.6)?  in that case i'm happy but
would like to see the example SDRF file updated to include this column.
to be MIAME compliant, it looks this column must be used (from the MIAME
checklist, caps added by me):

"Samples used, extract preparation and labeling:=20

The origin of each biological sample (e.g., name of the organism, the
PROVIDER of the sample) and its characteristics (e.g., gender, age,
developmental stage, strain, or disease state)."

> As you wish, although I'm quite sure we can't enforce this=20
> when dealing=20
> with MAGE-TAB documents generated by end users.

much appreciated.  if the examples and the tools that generate templates
use this extension, that will help the situation.  in general, people
are used to this, both Affymetrix and GenePix have standard extensions
for their files.

> I was under the impression that automated pipeline systems=20
> would be using=20
> MAGEv2-ML;=20

you don't give yourself and ArrayExpress enough credit here.  if it is
possible to get a paper published by submitting the data and annotation
using MAGETAB, people are going to expect their GeneExpression
applications to take these very same file formats.

> I'd rather not have to reinvent the=20
> definition of tab-delimited format; as you've seen, things=20
> get complicated=20
> quite quickly, to the point at which the name of the format=20
> *deceptively*=20
> implies simplicity. =20

my point is that allowing quotes means that MAGETAB is no longer a
simple tab-delimited file, it requires a state machine to parse it
because " becomes a called out character along with the tab.  are tabs
allowed between the quotes?  i would think a user would expect that they
are.

the fact that excel puts quotes around its fields is an artifice of
spread-sheet, not tab-delimited format.

but as i said i am perfectly willing to go with the consensus here.

> Even so, the MAQC experiment (E-TABM-132 in AE) was quite=20
> successfully=20
> coded on a single Tab2MAGE spreadsheet. I'm not clear why=20
> this is not an=20
> option for other such data sets.

the curious thing about this is that BioMaterial 2 and 26 for the
primary human samples do have BioSourceProvider attributes provided.
also, in principle the experiment isn't MIAME compliant because the rat
samples do not have their PROVIDER listed.

cheers,
michael

> -----Original Message-----
> From: Tim Rayner [mailto:ra...@eb...]=20
> Sent: Thursday, February 01, 2007 3:21 AM
> To: Miller, Michael D (Rosetta)
> Cc: MGED-mage; mged-mage2
> Subject: RE: [Mged-mage2] [Mged-mage] Collated comments on=20
> MAGETAB spec
>=20
>=20
> Hi Michael,
>=20
> Thanks for your comments. I have attempted to address them below:
>=20
> >> 2. Regarding adding a mandatory .idf extension for the IDF
> >> file, I think
> >> this is not necessary, since it's pretty easy to design a=20
> submission
> >> system which would track the IDF without an extension
> >
> > this is not true of automated pipeline systems, which is a=20
> huge use case
> > (as much or more gene expression data and annotation are loaded via
> > these pipelines).  in windows, anyway, it is very easy to=20
> set this up to
> > open automatically in excel or whatever spreadsheet program=20
> is desired.
>=20
> I was under the impression that automated pipeline systems=20
> would be using=20
> MAGEv2-ML; I'm very concerned that we're mixing up our use=20
> cases here.=20
> MAGE-TAB is not supposed to replace MAGEv2-ML, but complement=20
> it. It's a=20
> simple tabular format that bench scientists can use (in=20
> theory), *not* a=20
> wholesale replacement for the object modelling approach. I=20
> worry greatly=20
> that this has been forgotten in the rush to get this specification=20
> finished.
>=20
> > i would be happy with a recommendation that the file=20
> extension be 'idf'.
> > if this could be added to the 1st paragraph of section=20
> 3.1.1, that would
> > be great.
>=20
> As you wish, although I'm quite sure we can't enforce this=20
> when dealing=20
> with MAGE-TAB documents generated by end users.
>=20
> >> 3. Use of quotes in IDF and SDRF; while maybe they could be
> >> dispensed with
> >> in SDRF, they will be needed in IDF to allow e.g. newlines in
> >> protocol
> >> text (and believe me, users will want this!).
> >
> > as the addition of information on addition annotation files=20
> in the idf
> > was considered a late addition without time for comment, this is a
> > rather late addition, it was never part of the specification proper.
> > they can not be made optional, they either must be mandatory or not.
>=20
> I know of at least two well-developed parser modules for perl=20
> which allow=20
> quotes to be optional (Text::CSV_XS and Text::xSV). I have=20
> been assuming=20
> that similar parsers exist for Java and other languages.=20
> Tab2MAGE uses=20
> Text::CSV_XS and has had no problems with optional quoting.
>=20
> > i would like to see a handful of fields (like protocol descriptions)
> > where they are required and the rest where they must not be=20
> used but i
> > can live with them being mandatory in the IDF and not used=20
> in the SDRF.
>=20
> I think we need to use the same treatment for all the files;=20
> not only will=20
> that allow a uniform coding for parsers, but (critically) it will not=20
> confuse the users as much.
>=20
> > there also must be a provision for escaping quotes within the quoted
> > field, believe me, they will definitely occur (the escape can be as
> > simple as '\"' and, less likely, but possible, it will be the first
> > actual character of the field, which is why they must not=20
> be optional
> > for fields)
>=20
> I agree regarding the \" escape, and I've now added a note on that.=20
> However, a decent parser should be able to distinguish the following:
>=20
> <tab>"your text"<tab>  (" used to quote the field)
> <tab>"your" text<tab>  (" used within the content of the field)
>=20
> If the second quote in a pair is followed by a tab then that=20
> quote pair is=20
> quoting the field and is not part of the content. This is=20
> pretty much a=20
> solved problem on the perl side of things, although I'd definitely=20
> appreciate feedback if there aren't suitable Java parsers=20
> (this would,=20
> however, surprise me greatly). I'd rather not have to reinvent the=20
> definition of tab-delimited format; as you've seen, things=20
> get complicated=20
> quite quickly, to the point at which the name of the format=20
> *deceptively*=20
> implies simplicity. However, we're never going to be able to get=20
> bench-based end users to comply with a mandatory requirement for or=20
> against quoting here.
>=20
> Note that if it turns out there's a consensus against all of=20
> this then I=20
> can remove all the quotes from all the example files and the=20
> discussion in=20
> section 3.1.6. Any code I write will, however, handle quotes=20
> this way,=20
> because I know it works; the alternative does not. I added it to the=20
> specification because I believe that we can't avoid this in practice.
>=20
> >> 4. We agreed that the authority:namespace component of the
> >> resulting MAGE
> >> identifiers would be left to the implementation of any
> >> parser; while this
> >> precludes sharing e.g. Sources defined in MAGE-TAB documents,
> >> this is a
> >> relatively minor use case. Again, this could be revisited for a 1.1
> >> specification.
> >
> > this is not a relatively minor use case for many who would=20
> want to use
> > MAGETAB--it may be for ArrayExpress right now but it is a common use
> > case for investigation on how to organize a microarray experiment
> > (MAQC!!!).  the sharing of sources also occurs for a=20
> variety of other
> > reasons.
>=20
> I think I should have said "a relatively minor use case for=20
> MAGE-TAB".=20
> Again, what do bench biologists know of authorities and namespaces?=20
> MAGE-TAB is not a replacement for MAGEv2-ML, which I'd have=20
> thought would=20
> be a far more natural tool for large coordinated studies such=20
> as these.=20
> Even so, the MAQC experiment (E-TABM-132 in AE) was quite=20
> successfully=20
> coded on a single Tab2MAGE spreadsheet. I'm not clear why=20
> this is not an=20
> option for other such data sets.
>=20
> > this, i thought would be a minor addition to the IDF file=20
> to define the
> > default naming authority.  this is not a parser issue--a parser is
> > developed in accordance to the specification.  this would=20
> be very bad to
> > leave until 1.1, it will cause valuable biological information to be
> > lost.
>=20
> We decided very early on that the scope of the identifiers in=20
> the MAGE-TAB=20
> document is limited to within that document. However, if there's a=20
> consensus from everyone that this now needs to be changed,=20
> then of course=20
> we can change it. Given that the deadline for comments was 13=20
> days ago,=20
> though, with all these suggestions made after that deadline,=20
> I'm somewhat=20
> disappointed that we're still having this discussion.
>=20
> Best regards,
>=20
> Tim
>=20
>=20
>=20
> >
> >
> >> -----Original Message-----
> >> From: mge...@li...
> >> [mailto:mge...@li...] On Behalf
> >> Of Tim Rayner
> >> Sent: Wednesday, January 31, 2007 5:46 AM
> >> To: 'MGED-mage'; mged-mage2
> >> Subject: Re: [Mged-mage2] [Mged-mage] Collated comments on
> >> MAGETAB spec
> >>
> >>
> >> Hi,
> >>
> >> We've discussed the various comments on the previous version of the
> >> MAGE-TAB spec (Jan 8), and it appears that the consensus from
> >> ArrayExpress
> >> is as follows:
> >>
> >> 1. Since we need to get a version 1.0 specification=20
> finalised so that
> >> implementation deadlines are met, we feel that full support
> >> for external
> >> annotation files should be deferred to version 1.1. This will
> >> allow for
> >> more complete discussion of the requirements, in particular
> >> in light of
> >> any considerations from the FuGE crowd. In the meantime a=20
> minor note
> >> has been added to section 3.1.1 regarding suggested use of a
> >> Comment[] tag
> >> to support these in the meantime.
> >>
> >> 2. Regarding adding a mandatory .idf extension for the IDF
> >> file, I think
> >> this is not necessary, since it's pretty easy to design a=20
> submission
> >> system which would track the IDF without an extension (e.g.,
> >> the current
> >> Tab2MAGE submissions system already does this, in effect).
> >> Additionally, a
> >> new file extension would have to be mapped to Excel or
> >> OpenOffice by the
> >> end user for it to be any use to them (I believe this is=20
> true of both
> >> Windows and Mac). This may not be a huge deal, but it's
> >> another barrier to
> >> the casual user. Such mapping does not always guarantee that
> >> a document
> >> opens in the desired application either (OpenOffice, I'm looking at
> >> you...).
> >>
> >> 3. Use of quotes in IDF and SDRF; while maybe they could be
> >> dispensed with
> >> in SDRF, they will be needed in IDF to allow e.g. newlines in
> >> protocol
> >> text (and believe me, users will want this!). The original Tab2MAGE
> >> implementation didn't allow fields to be quoted like this=20
> to preserve
> >> special characters (i.e., newlines and tabs), and it was awful. An
> >> additional advantage to using quotes is that "text" date
> >> fields such as
> >> "2007-01-31" will be preserved by spreadsheet software, while
> >> 2007-01-31
> >> is often corrupted by such "helpful" applications. I have=20
> added a new
> >> section (3.1.6) which briefly discusses use of quotes to=20
> escape data
> >> fields. On the bright side, this format tends to be the default
> >> for "tab-delimited" export from Excel and OpenOffice in any case.
> >>
> >> 4. We agreed that the authority:namespace component of the
> >> resulting MAGE
> >> identifiers would be left to the implementation of any
> >> parser; while this
> >> precludes sharing e.g. Sources defined in MAGE-TAB documents,
> >> this is a
> >> relatively minor use case. Again, this could be revisited for a 1.1
> >> specification.
> >>
> >> 5. Regarding Junmin's comment about array design accessions,
> >> it is true
> >> that submissions to us will be using ArrayExpress accessions
> >> exclusively
> >> (at least for the forseeable future) but this is not
> >> necessarily true of
> >> other users, e.g. those downloading and distributing MAGE-TAB
> >> documents
> >> from us.
> >>
> >> 6. As discussed, the specification does indeed allow SDRFs to
> >> be split (on
> >> any "Name" column) into as many sub-SDRF documents as necessary.
> >>
> >> I've made the other modifications suggested to the list and
> >> put up a new
> >> specification document (Jan 31) here:
> >>
> >>=20
> http://www.ebi.ac.uk/systems-srv/mp/file-exchange/MAGE-TABv1.0.tar.gz
> >>
> >> Unless there are serious arguments to the contrary, I
> >> personally will be
> >> treating this as a finalised version 1.0 specification.
> >> Thanks very much
> >> for all your comments,
> >>
> >> Tim
> >>
> >>
> >> --
> >> Tim Rayner, Ph.D.
> >> Scientific Database Curator
> >> Microarray Informatics Team
> >> European Bioinformatics Institute
> >>
> >>
> >> --------------------------------------------------------------
> >> -----------
> >> Take Surveys. Earn Cash. Influence the Future of IT
> >> Join SourceForge.net's Techsay panel and you'll get the
> >> chance to share your
> >> opinions on IT & business topics through brief surveys -=20
> and earn cash
> >> http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge
> > &CID=3DDEVDEV
> > _______________________________________________
> > Mged-MAGE2 mailing list
> > Mge...@li...
> > https://lists.sourceforge.net/lists/listinfo/mged-mage2
> >
> >
> >
>=20
>=20