Re: [Miase-discuss] MIASE paper

[Mike C. <m.c...@au...>]

Dagmar, 

For 3) off the top of my head I would think 'reproducible' could mean:

Starting with the MIASE description, and access to the software and hardware
required to implement the algorithm(s) suggested therein, can I reproduce
the 'desired output' within a 'reasonable' level of tolerance. 'Desired
output' might be a data set, a figure, or some relationship derived from
those (for stochastic simulations, maybe the 'desired output' is a
particular pattern or trend line emerging from a figure plotted from data
from multiple runs).

For 5) I also think the two levels might be better as two 'use cases',
perhaps towards opposing ends of a continuum, more in line with the idea
expressed in:

"Information on elements of the simulation system that may have minor but
non-substantive effects on simulation outcome, such as the CPU or operating
system used, could also be provided but are outside the scope of MIASE and
are not considered part of the minimum requirements for reproducibility."

My 2c,


-----Original Message-----
From: Dagmar Köhn [mailto:da...@eb...] 
Sent: Friday, 8 May 2009 8:49 p.m.
To: mia...@li...; p.n...@au...;
p.h...@au...; Andrew Miller; j.l...@au...;
r.b...@au...; Mike Cooling; Catherine Lloyd;
e.c...@au...; David Nickerson; ala...@dp...;
db...@mc...; dcook@u.washington.edu; Stefan Hoops; bza...@in...;
hsauro@u.washington.edu; ped...@ma...; Dominic Tolle;
Vijayalakshmi Chelliah
Subject: MIASE paper

Dear all,

there have been some improvements on the paper (hopefully!), thanks to many
helpful comments by Mike Cooling and Frank Bergman (thanks!). I have
attached an updated version of the paper draft for you (odt & pdf).

Besides some remaining smaller issues, there are also some more striking
ones which need further discussions. Summarised further down and waiting for
your comments. Surely, I did forget to put issues, so feel free to open the
discussion on them!

If you have time to go through the paper (again), I'd say that the
motivation, "what is not in the scope of miase", the repressilator example
(which will be enhanced according to a suggestion by MC towards describing
post-processing also), and discussion sections are fine for now. But the
rules and the sections belonging to them need to be read again (especially
"Information on the model" / "- simulation")!

Best,
Dagmar

issues & required opinions:

1) future direction: Do you want me to include some comments on other
existing approaches for the description of simulation experiments or is it
irrelevant as we are talking about MIs?

2) the glossary: ... needs to be filled with definitions for simulation,
experiment, MIASE-compliance, (model, model instance). Suggestions welcome.

3) results: the notion of correctness/reproducibility.
In the paper we currently say that
"The Minimum Information About a Simulation Experiment (MIASE) is a set of
guidelines that serve as a recommendation for information to be provided
together with models in order to enable fast, simple and reproducible
simulation."
But we do not define nor say what "reproduce"/"reproducible" means to us.

Also, we say that
"The description of a simulation can lead to different levels of
correctness. To be MIASE compliant, a description should reproduce the
result of an experiment in a correct manner."
But we do not specify what "correct" means for us (When is a simulation
result actually really corresponding to an earlier result?, thinking also
about stochastic simulations).

This directly relates to the next issue:
5) Information on the simulation: the MIASE-compliance level First, I
thought it was a good idea to have different levels of MIASE compliance in
order to make a difference between levels of details of the simulation
description. But I think this is contra-productive as we call the whole
thing an "MI" which should either be minimum or not!? So I suppose we should
agree on the MI and not on different levels of it.
However, there was the idea of using 2 different MIASE levels (which we
could re-use as 2 different use cases where MIASE-compliant descriptions
differ a lot, but both are "correct"):
[comment Mike Cooling]
"I suspect there could be two levels of MIASE information (this is
orthogonal to what Mike Hucka was suggesting as in your point on progressive
guidelines). One might be conceptual and mentions the algorithm used (not
implementation), and general processes to achieve the example result. The
second level might be more an inventory of how the example result was
actually achieved, listing the implementation of said algorithm(s).
The first level would be used by those wanting to provide some transparency
and quality assessment, and allow those without the exact same technology
some kind of shot at reproducing the same (or similar) results. The second
('implementation') level would provide total transparency which could be
become useful if a) one doesn't have much time and just wants to exactly
reproduce an example output or b) one tries to use one's own technology to
produce the example result via the first level but for some reason can't -
the exact details of an example run would be available to help the keen
track down the difference between their output and the example. "

6) MIASE rules: please, could everyone go through them again and make sure
they all make sense, they are "complete" and correspond to what was said in
Auckland.

7) open comments from the Auckland meeting There are some of Nicolas'
comments on the Auckland meeting left that I did not yet include in the
paper, because basically I was not sure what they meant to say. Those are:
- (in the rules section, including) Geometry, Space discretization, Physics
and Constitutive parameters (PH)
- (in the guidelines section, talking about MIASE applicability) Could
domain-specific information be encoded in external ontology? (PH)
- (in information on the simulation) "We need to mention procedures to
ensure MIASE-compliance. This may be a problem in the case of proprietary
code. A repository or a registry of simulation codes may be a way.” -> this
goes to the compliance discussion actually!

8) KiSAO
We had a bit of discussion on whether KiSAO at its current state could
satisfy the needs for MIASE-compliant algorithm specification. But I think
we found a work around for the example given in the paper and the rest of
discussion about KiSAO should go to miase-discuss... Frank!?