[Pyphant-devel] Zip files in fmf / many fmf files belonging together

[Klaus Z. <kla...@fm...>]

Hi all,

Rolf wrote:
> Andreas wrote: 
> > But on the other hand aggregation of FMF-files contained within
> > a ZIP-archive is an important feature, which should be formally 
> > introduced in FMF 1.2. But this feature does need SampleContainers.
> I don't get the point on the issue with the zip-files.
> [...]

> From my simple point of view I would see three handlings/issues:
> 
> You get a bunch of fmf-files which where stored in one zip-file
> to save bandwidth (and maybe help handling)
> 
> You get a bunch of files stored in an zip file and some of them
> where fmf-files. 
> 
> An third approach would be to store the data compressed while
> keeping the meta-information human-readable
> 
> IMHO the first two issues would be part of the application
> and not the standard. The third one would be something which 
> my be interesting, but we should be aware that it breaks some
> point of the standard: the human readability of data.

Just to clarify, here is an example where we use zip files right now:
One of our customers maps a physical sample with a small light beam,
getting a spectrum for every point on an arbitrary grid. Each such
spectrum is saved in a fmf file and the set of all files making up the
sample are zipped together. Then on loading *of the zip file* the fmf
parser aggregates the data from all the fmf files, using some heuristics
to generate a sample container with the necessary redundancy mostly
removed. For example, if a key value pair in a general section appears
identically in all files it will end up as an attribute of the sample
container, whereas it will result in a column if it changes across the
fmf files. This means of course, that if some parameter is identical
across all files by accident, it will not produce the correct column,
but a surprising attribute. 
This happened to us for example when we extracted a certain row of
pixels from the sample. In that case all files had the same y coordinate
which messed up our pyphant recipe for the expected why column did no
longer exist in the respective sample containers.

This is what Andreas had in mind, when he wrote about zip files: There
should be some specification allowing for general attributes and so on.

So what do you think:
1) Do we need a specification for aggregate data?
2) Is zip a good means for that?
3) Can we get proper inspiration from prior art? What about jar files,
eggs?

Best,
Klaus