Re: [Miase-discuss] MIASE paper - reproducibility

[Dagmar K. <da...@eb...>]

Mike Cooling wrote:
> Dagmar, 
>
> For 3) off the top of my head I would think 'reproducible' could mean:
>
> Starting with the MIASE description, and access to the software and hardware
> required to implement the algorithm(s) suggested therein, can I reproduce
> the 'desired output' within a 'reasonable' level of tolerance. 'Desired
> output' might be a data set, a figure, or some relationship derived from
> those (for stochastic simulations, maybe the 'desired output' is a
> particular pattern or trend line emerging from a figure plotted from data
> from multiple runs).
>   

OK, I agree on the desired output.
Question is how detailed we would define the 'level of tolerance'. Maybe 
we want to leave it as "vague" as this and let the user decide?

Regarding the reproducibility...
Well, being a bit picky, and because we had kind of a related discussion 
with Frank and Nicolas, I checked the term reproducibility, which is 
often defined as: "the ability of a test or experiment to be accurately 
reproduced, or replicated, by someone else working independently." 
Re-occurring in all the definitions is the fact that in order to be 
reproducible, an experiment has to be run by several groups/ under 
different lab conditions.
I am not sure, but tend to think that "someone else" might not only 
include another user on the same simulation tool, but also a different 
simulation/computational environment?!

Maybe we want to avoid the discussion on reproducibility and just state 
that the repetition of experiments is enabled, which is easier to 
achieve as repeatable does not demand different conditions per 
definition. ("Repeatability is the variation in measurements taken by a 
single person or instrument on the same item and under the same 
conditions. A measurement may be said to be repeatable when this 
variation is smaller than some agreed limit.")

Doing so, we will say that following the MIASE guidelines, experiments 
can be repeated. We do not state though that they are reproducible. As a 
consequence exchanging them between simulation tools will not be 
guaranteed to be possible.

Is that limitation how far we want to go? Or do we want to include the 
reproducibility statement as well? In that case, we probably have to 
define the "agreed limit" or "level of tolerance" for a reproduced 
simulation result for ourselves and the readers.

BTW, I am in favor of including the reproducibility, as this is what 
makes MIASE most exciting ... :-)

Dagmar

> For 5) I also think the two levels might be better as two 'use cases',
> perhaps towards opposing ends of a continuum, more in line with the idea
> expressed in:
>
> "Information on elements of the simulation system that may have minor but
> non-substantive effects on simulation outcome, such as the CPU or operating
> system used, could also be provided but are outside the scope of MIASE and
> are not considered part of the minimum requirements for reproducibility."
>
> My 2c,
>
>
> -----Original Message-----
> From: Dagmar Köhn [mailto:da...@eb...] 
> Sent: Friday, 8 May 2009 8:49 p.m.
> To: mia...@li...; p.n...@au...;
> p.h...@au...; Andrew Miller; j.l...@au...;
> r.b...@au...; Mike Cooling; Catherine Lloyd;
> e.c...@au...; David Nickerson; ala...@dp...;
> db...@mc...; dcook@u.washington.edu; Stefan Hoops; bza...@in...;
> hsauro@u.washington.edu; ped...@ma...; Dominic Tolle;
> Vijayalakshmi Chelliah
> Subject: MIASE paper
>
> Dear all,
>
> there have been some improvements on the paper (hopefully!), thanks to many
> helpful comments by Mike Cooling and Frank Bergman (thanks!). I have
> attached an updated version of the paper draft for you (odt & pdf).
>
> Besides some remaining smaller issues, there are also some more striking
> ones which need further discussions. Summarised further down and waiting for
> your comments. Surely, I did forget to put issues, so feel free to open the
> discussion on them!
>
> If you have time to go through the paper (again), I'd say that the
> motivation, "what is not in the scope of miase", the repressilator example
> (which will be enhanced according to a suggestion by MC towards describing
> post-processing also), and discussion sections are fine for now. But the
> rules and the sections belonging to them need to be read again (especially
> "Information on the model" / "- simulation")!
>
> Best,
> Dagmar
>
> issues & required opinions:
>
> 1) future direction: Do you want me to include some comments on other
> existing approaches for the description of simulation experiments or is it
> irrelevant as we are talking about MIs?
>
> 2) the glossary: ... needs to be filled with definitions for simulation,
> experiment, MIASE-compliance, (model, model instance). Suggestions welcome.
>
> 3) results: the notion of correctness/reproducibility.
> In the paper we currently say that
> "The Minimum Information About a Simulation Experiment (MIASE) is a set of
> guidelines that serve as a recommendation for information to be provided
> together with models in order to enable fast, simple and reproducible
> simulation."
> But we do not define nor say what "reproduce"/"reproducible" means to us.
>
> Also, we say that
> "The description of a simulation can lead to different levels of
> correctness. To be MIASE compliant, a description should reproduce the
> result of an experiment in a correct manner."
> But we do not specify what "correct" means for us (When is a simulation
> result actually really corresponding to an earlier result?, thinking also
> about stochastic simulations).
>
> This directly relates to the next issue:
> 5) Information on the simulation: the MIASE-compliance level First, I
> thought it was a good idea to have different levels of MIASE compliance in
> order to make a difference between levels of details of the simulation
> description. But I think this is contra-productive as we call the whole
> thing an "MI" which should either be minimum or not!? So I suppose we should
> agree on the MI and not on different levels of it.
> However, there was the idea of using 2 different MIASE levels (which we
> could re-use as 2 different use cases where MIASE-compliant descriptions
> differ a lot, but both are "correct"):
> [comment Mike Cooling]
> "I suspect there could be two levels of MIASE information (this is
> orthogonal to what Mike Hucka was suggesting as in your point on progressive
> guidelines). One might be conceptual and mentions the algorithm used (not
> implementation), and general processes to achieve the example result. The
> second level might be more an inventory of how the example result was
> actually achieved, listing the implementation of said algorithm(s).
> The first level would be used by those wanting to provide some transparency
> and quality assessment, and allow those without the exact same technology
> some kind of shot at reproducing the same (or similar) results. The second
> ('implementation') level would provide total transparency which could be
> become useful if a) one doesn't have much time and just wants to exactly
> reproduce an example output or b) one tries to use one's own technology to
> produce the example result via the first level but for some reason can't -
> the exact details of an example run would be available to help the keen
> track down the difference between their output and the example. "
>
> 6) MIASE rules: please, could everyone go through them again and make sure
> they all make sense, they are "complete" and correspond to what was said in
> Auckland.
>
> 7) open comments from the Auckland meeting There are some of Nicolas'
> comments on the Auckland meeting left that I did not yet include in the
> paper, because basically I was not sure what they meant to say. Those are:
> - (in the rules section, including) Geometry, Space discretization, Physics
> and Constitutive parameters (PH)
> - (in the guidelines section, talking about MIASE applicability) Could
> domain-specific information be encoded in external ontology? (PH)
> - (in information on the simulation) "We need to mention procedures to
> ensure MIASE-compliance. This may be a problem in the case of proprietary
> code. A repository or a registry of simulation codes may be a way.” -> this
> goes to the compliance discussion actually!
>
> 8) KiSAO
> We had a bit of discussion on whether KiSAO at its current state could
> satisfy the needs for MIASE-compliant algorithm specification. But I think
> we found a work around for the example given in the paper and the rest of
> discussion about KiSAO should go to miase-discuss... Frank!?
>
>
>
>