From: Miller, M. D (Rosetta) <Michael_Miller@Rosettabio.com> - 2007-02-05 16:18:47
|
hi tim, you've addressed my major concerns, and outside of any feedback on the " issue from others, i'm satisfied > I should perhaps also comment that I fully expect there to be=20 > a whole slew=20 > of things we've missed in 1.0 which will turn up during=20 > various peoples'=20 > implementations,=20 my main concern was about where the sources came from, so the Provider column makes me more than happy to wait until a 1.1 to expand on this and other issues. cheers, michael=20 > -----Original Message----- > From: Tim Rayner [mailto:ra...@eb...]=20 > Sent: Monday, February 05, 2007 6:18 AM > To: Miller, Michael D (Rosetta) > Cc: MGED-mage; mged-mage2 > Subject: RE: [Mged-mage2] [Mged-mage] Collated comments on=20 > MAGETAB spec >=20 >=20 > Hi Michael, >=20 > Thank you once again for your comments; answers inline below... >=20 > On Sat, 3 Feb 2007, Miller, Michael D (Rosetta) wrote: >=20 > > hi tim, > > > > thanks for your time. my initial comments came two weeks=20 > back and my > > main sticking point, where the Sources were obtained from,=20 > i thought was > > covered in the specification itself but when i was working=20 > out examples > > didn't find what i had seen before. i've taken another=20 > look and isn't > > this the "PROVIDER" column (section 3.3.6)? in that case=20 > i'm happy but > > would like to see the example SDRF file updated to include=20 > this column. > > to be MIAME compliant, it looks this column must be used=20 > (from the MIAME > > checklist, caps added by me): >=20 > Yes, that is indeed the intention. It doesn't provide quite the=20 > fine-grained control of identifiers and namespaces that might be=20 > desirable, of course. Note that I'm not adverse to adding=20 > (optional!) IDF=20 > tags "Investigation Namespace" and "Investigation Authority"=20 > or somesuch=20 > in 1.1, but I do think a wider discussion is needed. I've added a new=20 > Provider column to the example SDRF as requested. >=20 > I should perhaps also comment that I fully expect there to be=20 > a whole slew=20 > of things we've missed in 1.0 which will turn up during=20 > various peoples'=20 > implementations, and I've been envisioning a 1.1 specification sooner=20 > rather than later (maybe at the Stanford jamboree, or even=20 > sooner?). Now=20 > is definitely the time to be setting the agenda for such an update. >=20 > > "Samples used, extract preparation and labeling: > > > > The origin of each biological sample (e.g., name of the=20 > organism, the > > PROVIDER of the sample) and its characteristics (e.g., gender, age, > > developmental stage, strain, or disease state)." > > > >> As you wish, although I'm quite sure we can't enforce this > >> when dealing > >> with MAGE-TAB documents generated by end users. > > > > much appreciated. if the examples and the tools that=20 > generate templates > > use this extension, that will help the situation. in=20 > general, people > > are used to this, both Affymetrix and GenePix have standard=20 > extensions > > for their files. >=20 > Good point - I've also fixed that in the example files. >=20 > >> I was under the impression that automated pipeline systems > >> would be using > >> MAGEv2-ML; > > > > you don't give yourself and ArrayExpress enough credit=20 > here. if it is > > possible to get a paper published by submitting the data=20 > and annotation > > using MAGETAB, people are going to expect their GeneExpression > > applications to take these very same file formats. > > > >> I'd rather not have to reinvent the > >> definition of tab-delimited format; as you've seen, things > >> get complicated > >> quite quickly, to the point at which the name of the format > >> *deceptively* > >> implies simplicity. > > > > my point is that allowing quotes means that MAGETAB is no longer a > > simple tab-delimited file, it requires a state machine to parse it > > because " becomes a called out character along with the tab. >=20 > Yes, to an extent it depends on how "tab-delimited" is=20 > defined, and that=20 > seems to be at the crux of this discussion. If a spreadsheet app user=20 > exports "tab-delimited" text, for better or worse the default is to=20 > sprinkle it liberally with quotes around every text field. >=20 > > are tabs > > allowed between the quotes? i would think a user would=20 > expect that they > > are. >=20 > That would be the intent, yes. >=20 > > the fact that excel puts quotes around its fields is an artifice of > > spread-sheet, not tab-delimited format. > > > > but as i said i am perfectly willing to go with the consensus here. >=20 > I whole-heartedly agree (to both points). >=20 > >> Even so, the MAQC experiment (E-TABM-132 in AE) was quite > >> successfully > >> coded on a single Tab2MAGE spreadsheet. I'm not clear why > >> this is not an > >> option for other such data sets. > > > > the curious thing about this is that BioMaterial 2 and 26 for the > > primary human samples do have BioSourceProvider attributes provided. > > also, in principle the experiment isn't MIAME compliant=20 > because the rat > > samples do not have their PROVIDER listed. >=20 > I'll look into that, thanks. Most likely the information was=20 > simply not=20 > provided by the submitter. In the meantime the latest (and I=20 > truly hope,=20 > final this time) v1.0 spec is available from the same link as before. >=20 > Cheers, >=20 > Tim >=20 >=20 > > cheers, > > michael > > > >> -----Original Message----- > >> From: Tim Rayner [mailto:ra...@eb...] > >> Sent: Thursday, February 01, 2007 3:21 AM > >> To: Miller, Michael D (Rosetta) > >> Cc: MGED-mage; mged-mage2 > >> Subject: RE: [Mged-mage2] [Mged-mage] Collated comments on > >> MAGETAB spec > >> > >> > >> Hi Michael, > >> > >> Thanks for your comments. I have attempted to address them below: > >> > >>>> 2. Regarding adding a mandatory .idf extension for the IDF > >>>> file, I think > >>>> this is not necessary, since it's pretty easy to design a > >> submission > >>>> system which would track the IDF without an extension > >>> > >>> this is not true of automated pipeline systems, which is a > >> huge use case > >>> (as much or more gene expression data and annotation are=20 > loaded via > >>> these pipelines). in windows, anyway, it is very easy to > >> set this up to > >>> open automatically in excel or whatever spreadsheet program > >> is desired. > >> > >> I was under the impression that automated pipeline systems > >> would be using > >> MAGEv2-ML; I'm very concerned that we're mixing up our use > >> cases here. > >> MAGE-TAB is not supposed to replace MAGEv2-ML, but complement > >> it. It's a > >> simple tabular format that bench scientists can use (in > >> theory), *not* a > >> wholesale replacement for the object modelling approach. I > >> worry greatly > >> that this has been forgotten in the rush to get this specification > >> finished. > >> > >>> i would be happy with a recommendation that the file > >> extension be 'idf'. > >>> if this could be added to the 1st paragraph of section > >> 3.1.1, that would > >>> be great. > >> > >> As you wish, although I'm quite sure we can't enforce this > >> when dealing > >> with MAGE-TAB documents generated by end users. > >> > >>>> 3. Use of quotes in IDF and SDRF; while maybe they could be > >>>> dispensed with > >>>> in SDRF, they will be needed in IDF to allow e.g. newlines in > >>>> protocol > >>>> text (and believe me, users will want this!). > >>> > >>> as the addition of information on addition annotation files > >> in the idf > >>> was considered a late addition without time for comment, this is a > >>> rather late addition, it was never part of the=20 > specification proper. > >>> they can not be made optional, they either must be=20 > mandatory or not. > >> > >> I know of at least two well-developed parser modules for perl > >> which allow > >> quotes to be optional (Text::CSV_XS and Text::xSV). I have > >> been assuming > >> that similar parsers exist for Java and other languages. > >> Tab2MAGE uses > >> Text::CSV_XS and has had no problems with optional quoting. > >> > >>> i would like to see a handful of fields (like protocol=20 > descriptions) > >>> where they are required and the rest where they must not be > >> used but i > >>> can live with them being mandatory in the IDF and not used > >> in the SDRF. > >> > >> I think we need to use the same treatment for all the files; > >> not only will > >> that allow a uniform coding for parsers, but (critically)=20 > it will not > >> confuse the users as much. > >> > >>> there also must be a provision for escaping quotes within=20 > the quoted > >>> field, believe me, they will definitely occur (the escape=20 > can be as > >>> simple as '\"' and, less likely, but possible, it will be=20 > the first > >>> actual character of the field, which is why they must not > >> be optional > >>> for fields) > >> > >> I agree regarding the \" escape, and I've now added a note on that. > >> However, a decent parser should be able to distinguish the=20 > following: > >> > >> <tab>"your text"<tab> (" used to quote the field) > >> <tab>"your" text<tab> (" used within the content of the field) > >> > >> If the second quote in a pair is followed by a tab then that > >> quote pair is > >> quoting the field and is not part of the content. This is > >> pretty much a > >> solved problem on the perl side of things, although I'd definitely > >> appreciate feedback if there aren't suitable Java parsers > >> (this would, > >> however, surprise me greatly). I'd rather not have to reinvent the > >> definition of tab-delimited format; as you've seen, things > >> get complicated > >> quite quickly, to the point at which the name of the format > >> *deceptively* > >> implies simplicity. However, we're never going to be able to get > >> bench-based end users to comply with a mandatory requirement for or > >> against quoting here. > >> > >> Note that if it turns out there's a consensus against all of > >> this then I > >> can remove all the quotes from all the example files and the > >> discussion in > >> section 3.1.6. Any code I write will, however, handle quotes > >> this way, > >> because I know it works; the alternative does not. I added=20 > it to the > >> specification because I believe that we can't avoid this=20 > in practice. > >> > >>>> 4. We agreed that the authority:namespace component of the > >>>> resulting MAGE > >>>> identifiers would be left to the implementation of any > >>>> parser; while this > >>>> precludes sharing e.g. Sources defined in MAGE-TAB documents, > >>>> this is a > >>>> relatively minor use case. Again, this could be=20 > revisited for a 1.1 > >>>> specification. > >>> > >>> this is not a relatively minor use case for many who would > >> want to use > >>> MAGETAB--it may be for ArrayExpress right now but it is a=20 > common use > >>> case for investigation on how to organize a microarray experiment > >>> (MAQC!!!). the sharing of sources also occurs for a > >> variety of other > >>> reasons. > >> > >> I think I should have said "a relatively minor use case for > >> MAGE-TAB". > >> Again, what do bench biologists know of authorities and namespaces? > >> MAGE-TAB is not a replacement for MAGEv2-ML, which I'd have > >> thought would > >> be a far more natural tool for large coordinated studies such > >> as these. > >> Even so, the MAQC experiment (E-TABM-132 in AE) was quite > >> successfully > >> coded on a single Tab2MAGE spreadsheet. I'm not clear why > >> this is not an > >> option for other such data sets. > >> > >>> this, i thought would be a minor addition to the IDF file > >> to define the > >>> default naming authority. this is not a parser issue--a parser is > >>> developed in accordance to the specification. this would > >> be very bad to > >>> leave until 1.1, it will cause valuable biological=20 > information to be > >>> lost. > >> > >> We decided very early on that the scope of the identifiers in > >> the MAGE-TAB > >> document is limited to within that document. However, if there's a > >> consensus from everyone that this now needs to be changed, > >> then of course > >> we can change it. Given that the deadline for comments was 13 > >> days ago, > >> though, with all these suggestions made after that deadline, > >> I'm somewhat > >> disappointed that we're still having this discussion. > >> > >> Best regards, > >> > >> Tim > >> > >> > >> > >>> > >>> > >>>> -----Original Message----- > >>>> From: mge...@li... > >>>> [mailto:mge...@li...] On Behalf > >>>> Of Tim Rayner > >>>> Sent: Wednesday, January 31, 2007 5:46 AM > >>>> To: 'MGED-mage'; mged-mage2 > >>>> Subject: Re: [Mged-mage2] [Mged-mage] Collated comments on > >>>> MAGETAB spec > >>>> > >>>> > >>>> Hi, > >>>> > >>>> We've discussed the various comments on the previous=20 > version of the > >>>> MAGE-TAB spec (Jan 8), and it appears that the consensus from > >>>> ArrayExpress > >>>> is as follows: > >>>> > >>>> 1. Since we need to get a version 1.0 specification > >> finalised so that > >>>> implementation deadlines are met, we feel that full support > >>>> for external > >>>> annotation files should be deferred to version 1.1. This will > >>>> allow for > >>>> more complete discussion of the requirements, in particular > >>>> in light of > >>>> any considerations from the FuGE crowd. In the meantime a > >> minor note > >>>> has been added to section 3.1.1 regarding suggested use of a > >>>> Comment[] tag > >>>> to support these in the meantime. > >>>> > >>>> 2. Regarding adding a mandatory .idf extension for the IDF > >>>> file, I think > >>>> this is not necessary, since it's pretty easy to design a > >> submission > >>>> system which would track the IDF without an extension (e.g., > >>>> the current > >>>> Tab2MAGE submissions system already does this, in effect). > >>>> Additionally, a > >>>> new file extension would have to be mapped to Excel or > >>>> OpenOffice by the > >>>> end user for it to be any use to them (I believe this is > >> true of both > >>>> Windows and Mac). This may not be a huge deal, but it's > >>>> another barrier to > >>>> the casual user. Such mapping does not always guarantee that > >>>> a document > >>>> opens in the desired application either (OpenOffice, I'm=20 > looking at > >>>> you...). > >>>> > >>>> 3. Use of quotes in IDF and SDRF; while maybe they could be > >>>> dispensed with > >>>> in SDRF, they will be needed in IDF to allow e.g. newlines in > >>>> protocol > >>>> text (and believe me, users will want this!). The=20 > original Tab2MAGE > >>>> implementation didn't allow fields to be quoted like this > >> to preserve > >>>> special characters (i.e., newlines and tabs), and it was=20 > awful. An > >>>> additional advantage to using quotes is that "text" date > >>>> fields such as > >>>> "2007-01-31" will be preserved by spreadsheet software, while > >>>> 2007-01-31 > >>>> is often corrupted by such "helpful" applications. I have > >> added a new > >>>> section (3.1.6) which briefly discusses use of quotes to > >> escape data > >>>> fields. On the bright side, this format tends to be the default > >>>> for "tab-delimited" export from Excel and OpenOffice in any case. > >>>> > >>>> 4. We agreed that the authority:namespace component of the > >>>> resulting MAGE > >>>> identifiers would be left to the implementation of any > >>>> parser; while this > >>>> precludes sharing e.g. Sources defined in MAGE-TAB documents, > >>>> this is a > >>>> relatively minor use case. Again, this could be=20 > revisited for a 1.1 > >>>> specification. > >>>> > >>>> 5. Regarding Junmin's comment about array design accessions, > >>>> it is true > >>>> that submissions to us will be using ArrayExpress accessions > >>>> exclusively > >>>> (at least for the forseeable future) but this is not > >>>> necessarily true of > >>>> other users, e.g. those downloading and distributing MAGE-TAB > >>>> documents > >>>> from us. > >>>> > >>>> 6. As discussed, the specification does indeed allow SDRFs to > >>>> be split (on > >>>> any "Name" column) into as many sub-SDRF documents as necessary. > >>>> > >>>> I've made the other modifications suggested to the list and > >>>> put up a new > >>>> specification document (Jan 31) here: > >>>> > >>>> > >>=20 > http://www.ebi.ac.uk/systems-srv/mp/file-exchange/MAGE-TABv1.0.tar.gz > >>>> > >>>> Unless there are serious arguments to the contrary, I > >>>> personally will be > >>>> treating this as a finalised version 1.0 specification. > >>>> Thanks very much > >>>> for all your comments, > >>>> > >>>> Tim > >>>> > >>>> > >>>> -- > >>>> Tim Rayner, Ph.D. > >>>> Scientific Database Curator > >>>> Microarray Informatics Team > >>>> European Bioinformatics Institute > >>>> > >>>> > >>>> -------------------------------------------------------------- > >>>> ----------- > >>>> Take Surveys. Earn Cash. Influence the Future of IT > >>>> Join SourceForge.net's Techsay panel and you'll get the > >>>> chance to share your > >>>> opinions on IT & business topics through brief surveys - > >> and earn cash > >>>> = http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge > >>> &CID=3DDEVDEV > >>> _______________________________________________ > >>> Mged-MAGE2 mailing list > >>> Mge...@li... > >>> https://lists.sourceforge.net/lists/listinfo/mged-mage2 > >>> > >>> > >>> > >> > >> > > > > >=20 >=20 |