Re: [Miase-discuss] MIASE paper - reproducibility

[Richard A. <ra...@st...>]

Just on the reproducibility/repeatability issue.

I did a quick survey of  wet-lab standards are, looking in  
instructions for authors, and a quick survey of other Mibbi projects.

Looking at 4 top experimental journals:

Cell,
Nature MSB
Journal of Cell Biology
EMBO Journal

the only requirement for 'Material & Methods' sections are that   
sufficient detail is given so that
'all experimental procedures can be repeated by others'
i.e., no mention of reproducing results.

And a quick survey of 4  Mibbi projects: Misfishie, Miape-GE ,Mimix and Miare

The first three have goals of enabling understanding/ability to repeat  
experiment, but only Miare claims as its goal
'Minimum Information About an RNAi Experiment (MIARE) is a set of  
reporting guidelines which describes the minimum information that  
should be reported about an RNAi experiment to enable the unambiguous  
interpretation and REPRODUCTION of the results'(my caps)

The Miare spec has excruciating detail about all aspects of an RNAi  
experiment.

So, if we want to claim a Miase file aims to allow results to be  
reproduced, we would need to include lots of detail. Or, is there a  
fundamental distinction between computer simulations and experiment,  
that simulations are inherently more reproducible and thus do not need  
such level of detail.

Or, should the onus be on the curator of the miase file, who could  
check how robust the results were, using different software, different  
OS etc, and come up with some sort of 'reproducibility index', ranging  
from 'these results are only achieved using MyGreatSoftware version  
2.3.1beta on MacOSX10.5' to 'these results are reproducible using any  
standard implementation of algorithm X'. But, this would be quite a  
large burden on curators/model builders to evaluate the reproducibility.

Richard







> Mike Cooling wrote:
>> Dagmar,
>>
>> For 3) off the top of my head I would think 'reproducible' could mean:
>>
>> Starting with the MIASE description, and access to the software and hardware
>> required to implement the algorithm(s) suggested therein, can I reproduce
>> the 'desired output' within a 'reasonable' level of tolerance. 'Desired
>> output' might be a data set, a figure, or some relationship derived from
>> those (for stochastic simulations, maybe the 'desired output' is a
>> particular pattern or trend line emerging from a figure plotted from data
>> from multiple runs).
>>
>
> OK, I agree on the desired output.
> Question is how detailed we would define the 'level of tolerance'. Maybe
> we want to leave it as "vague" as this and let the user decide?
>
> Regarding the reproducibility...
> Well, being a bit picky, and because we had kind of a related discussion
> with Frank and Nicolas, I checked the term reproducibility, which is
> often defined as: "the ability of a test or experiment to be accurately
> reproduced, or replicated, by someone else working independently."
> Re-occurring in all the definitions is the fact that in order to be
> reproducible, an experiment has to be run by several groups/ under
> different lab conditions.
> I am not sure, but tend to think that "someone else" might not only
> include another user on the same simulation tool, but also a different
> simulation/computational environment?!
>
> Maybe we want to avoid the discussion on reproducibility and just state
> that the repetition of experiments is enabled, which is easier to
> achieve as repeatable does not demand different conditions per
> definition. ("Repeatability is the variation in measurements taken by a
> single person or instrument on the same item and under the same
> conditions. A measurement may be said to be repeatable when this
> variation is smaller than some agreed limit.")
>
> Doing so, we will say that following the MIASE guidelines, experiments
> can be repeated. We do not state though that they are reproducible. As a
> consequence exchanging them between simulation tools will not be
> guaranteed to be possible.
>
> Is that limitation how far we want to go? Or do we want to include the
> reproducibility statement as well? In that case, we probably have to
> define the "agreed limit" or "level of tolerance" for a reproduced
> simulation result for ourselves and the readers.
>
> BTW, I am in favor of including the reproducibility, as this is what
> makes MIASE most exciting ... :-)
>
> Dagmar
>
>> For 5) I also think the two levels might be better as two 'use cases',
>> perhaps towards opposing ends of a continuum, more in line with the idea
>> expressed in:
>>
>> "Information on elements of the simulation system that may have minor but
>> non-substantive effects on simulation outcome, such as the CPU or operating
>> system used, could also be provided but are outside the scope of MIASE and
>> are not considered part of the minimum requirements for reproducibility."
>>
>> My 2c,
>>
>>
>> -----Original Message-----
>> From: Dagmar Köhn [mailto:da...@eb...]
>> Sent: Friday, 8 May 2009 8:49 p.m.
>> To: mia...@li...; p.n...@au...;
>> p.h...@au...; Andrew Miller; j.l...@au...;
>> r.b...@au...; Mike Cooling; Catherine Lloyd;
>> e.c...@au...; David Nickerson; ala...@dp...;
>> db...@mc...; dcook@u.washington.edu; Stefan Hoops; bza...@in...;
>> hsauro@u.washington.edu; ped...@ma...; Dominic Tolle;
>> Vijayalakshmi Chelliah
>> Subject: MIASE paper
>>
>> Dear all,
>>
>> there have been some improvements on the paper (hopefully!), thanks to many
>> helpful comments by Mike Cooling and Frank Bergman (thanks!). I have
>> attached an updated version of the paper draft for you (odt & pdf).
>>
>> Besides some remaining smaller issues, there are also some more striking
>> ones which need further discussions. Summarised further down and waiting for
>> your comments. Surely, I did forget to put issues, so feel free to open the
>> discussion on them!
>>
>> If you have time to go through the paper (again), I'd say that the
>> motivation, "what is not in the scope of miase", the repressilator example
>> (which will be enhanced according to a suggestion by MC towards describing
>> post-processing also), and discussion sections are fine for now. But the
>> rules and the sections belonging to them need to be read again (especially
>> "Information on the model" / "- simulation")!
>>
>> Best,
>> Dagmar
>>
>> issues & required opinions:
>>
>> 1) future direction: Do you want me to include some comments on other
>> existing approaches for the description of simulation experiments or is it
>> irrelevant as we are talking about MIs?
>>
>> 2) the glossary: ... needs to be filled with definitions for simulation,
>> experiment, MIASE-compliance, (model, model instance). Suggestions welcome.
>>
>> 3) results: the notion of correctness/reproducibility.
>> In the paper we currently say that
>> "The Minimum Information About a Simulation Experiment (MIASE) is a set of
>> guidelines that serve as a recommendation for information to be provided
>> together with models in order to enable fast, simple and reproducible
>> simulation."
>> But we do not define nor say what "reproduce"/"reproducible" means to us.
>>
>> Also, we say that
>> "The description of a simulation can lead to different levels of
>> correctness. To be MIASE compliant, a description should reproduce the
>> result of an experiment in a correct manner."
>> But we do not specify what "correct" means for us (When is a simulation
>> result actually really corresponding to an earlier result?, thinking also
>> about stochastic simulations).
>>
>> This directly relates to the next issue:
>> 5) Information on the simulation: the MIASE-compliance level First, I
>> thought it was a good idea to have different levels of MIASE compliance in
>> order to make a difference between levels of details of the simulation
>> description. But I think this is contra-productive as we call the whole
>> thing an "MI" which should either be minimum or not!? So I suppose we should
>> agree on the MI and not on different levels of it.
>> However, there was the idea of using 2 different MIASE levels (which we
>> could re-use as 2 different use cases where MIASE-compliant descriptions
>> differ a lot, but both are "correct"):
>> [comment Mike Cooling]
>> "I suspect there could be two levels of MIASE information (this is
>> orthogonal to what Mike Hucka was suggesting as in your point on progressive
>> guidelines). One might be conceptual and mentions the algorithm used (not
>> implementation), and general processes to achieve the example result. The
>> second level might be more an inventory of how the example result was
>> actually achieved, listing the implementation of said algorithm(s).
>> The first level would be used by those wanting to provide some transparency
>> and quality assessment, and allow those without the exact same technology
>> some kind of shot at reproducing the same (or similar) results. The second
>> ('implementation') level would provide total transparency which could be
>> become useful if a) one doesn't have much time and just wants to exactly
>> reproduce an example output or b) one tries to use one's own technology to
>> produce the example result via the first level but for some reason can't -
>> the exact details of an example run would be available to help the keen
>> track down the difference between their output and the example. "
>>
>> 6) MIASE rules: please, could everyone go through them again and make sure
>> they all make sense, they are "complete" and correspond to what was said in
>> Auckland.
>>
>> 7) open comments from the Auckland meeting There are some of Nicolas'
>> comments on the Auckland meeting left that I did not yet include in the
>> paper, because basically I was not sure what they meant to say. Those are:
>> - (in the rules section, including) Geometry, Space discretization, Physics
>> and Constitutive parameters (PH)
>> - (in the guidelines section, talking about MIASE applicability) Could
>> domain-specific information be encoded in external ontology? (PH)
>> - (in information on the simulation) "We need to mention procedures to
>> ensure MIASE-compliance. This may be a problem in the case of proprietary
>> code. A repository or a registry of simulation codes may be a way.” -> this
>> goes to the compliance discussion actually!
>>
>> 8) KiSAO
>> We had a bit of discussion on whether KiSAO at its current state could
>> satisfy the needs for MIASE-compliant algorithm specification. But I think
>> we found a work around for the example given in the paper and the rest of
>> discussion about KiSAO should go to miase-discuss... Frank!?
>>
>>
>>
>>
>
>
> ------------------------------------------------------------------------------
> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
> production scanning environment may not be a perfect world - but thanks to
> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
> Series Scanner you'll get full speed at 300 dpi even with all image
> processing features enabled. http://p.sf.net/sfu/kodak-com
> _______________________________________________
> Miase-discuss mailing list
> Mia...@li...
> https://lists.sourceforge.net/lists/listinfo/miase-discuss
>
>



-- 
Dr Richard Adams
Senior Software Developer,
Computational Systems Biology Group,
University of Edinburgh
Tel: 0131 650 8285
email : ric...@ed...

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.