Re: [Psidev-ms-dev] Separate binary file for very large data sets?

[Coleman, M. <MK...@St...>]

If by "compressed" you mean "base64-encoded", I think it's important to =
use the latter term, to avoid giving the wrong impression.  As far as I =
know, compression is not a feature--nor a goal--of mzData.

For what it's worth, I encountered my first mzData file in a work =
situation this week.  It's 2.7 times as large as the corresponding ms2 =
file.

Mike



> -----Original Message-----
> From: psi...@li...=20
> [mailto:psi...@li...] On=20
> Behalf Of Brian Pratt
> Sent: Friday, April 20, 2007 2:02 PM
> To: 'Andreas R=F6mpp'; psi...@li...
> Subject: Re: [Psidev-ms-dev] Separate binary file for very=20
> large data sets?
>=20
>=20
> I wonder if it wouldn't make as much sense to treat the=20
> mzData file as the
> "binary file" and come up with a sort of summary schema of=20
> your own that
> could point into the mzData file.  You'd get maximum reuse of=20
> community
> source code that way.
>=20
> But first, I'd say try it with straight-up mzData with=20
> compressed peak lists
> and see if you really need to go to the bother of a separate=20
> file.  I'm
> guessing you'll be pleasantly surprised.  Plus, I really,=20
> really dislike the
> use of interdependent files - one or the other is forever=20
> getting out of
> synch, lost, renamed, etc.
>=20
> Hope this helps,
>=20
> Brian Pratt
> www.insilicos.com
>=20
> -----Original Message-----
> From: psi...@li...
> [mailto:psi...@li...] On=20
> Behalf Of Andreas
> R=F6mpp
> Sent: Friday, April 20, 2007 8:45 AM
> To: psi...@li...;
> And...@an...
> Subject: [Psidev-ms-dev] Separate binary file for very large=20
> data sets?
>=20
> Hello everybody,
>=20
> We develop software for imaging mass spectrometry in the=20
> framework of a
> project funded by the European Union. We intend to use=20
> dataXML as a standard
> format to exchange data between the different partner labs=20
> and also (as far
> as possible) as the internal data format for a joint=20
> processing software
> suite. However, we run into the problem of very large data=20
> sets which can
> easily exceed 1GB (e.g. 256
> *256 pixels with one high resolution mass spectrum each). Therefore we
> thought about storing the spectrum data=20
> ('MassToChargeRatioArray' and '=20
> 'IntensityArray') in a separate binary file. This would make=20
> data handling
> much faster and easier (e.g. when parsing the XML file). So instead of
> writing the binary data in the XML file we plan to include a link to a
> separate file (file location, start and end position of=20
> spectrum in binary
> file).
> This problem is somewhat similar to the already discussed=20
> issue of an index
> file.
> Would it be possible to include such an option (external=20
> binary file) into
> the dataXML standard?
>=20
> Best regards,
> Andreas
>=20
> --
> --------------------------------------------------------------
> --------------
> -------------
> Dr. Andreas Roempp
> Institute of Inorganic and Analytical Chemistry
> - Analytical Chemistry -
> Justus Liebig University Giessen
> Schubertstrasse 60, Build. 16
> D-35392 Giessen
> Germany
>=20
> phone:  +49-641-99 34802
> fax:    +49-641-99 34809
> email: And...@an...
> Internet: http://www.uni-giessen.de/analytik/
>=20
>=20
>=20
>=20
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by DB2 Express Download DB2=20
> Express C - the
> FREE version of DB2 express and take control of your XML. No=20
> limits. Just
> data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Psidev-ms-dev mailing list
> Psi...@li...
> https://lists.sourceforge.net/lists/listinfo/psidev-ms-dev
>=20
>=20
> --------------------------------------------------------------
> -----------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Psidev-ms-dev mailing list
> Psi...@li...
> https://lists.sourceforge.net/lists/listinfo/psidev-ms-dev
>=20