Re: [Miase-discuss] Dagmar Köhn: MIASE comments

[Beard, D. <db...@mc...>]

All,

I am a bit reluctant to join this discussion late, and worried that I have not caught all of the points that have been brought up. However, I have finally got to go through the paper and I have some comments.

I started marking up the paper by editing the text, but I realized that I was essentially rewriting every sentence. I concluded that maybe that is not the thing for me to do at this stage for two reasons: First, I don't think you want this paper to be "written by a committee". Rather, I think that it would be better written with a single voice, and since you have done a great job of getting this going, it will be great if that voice is yours. That said, I don't think that the paper is in shape yet for editing. So, I will limit my comments to the big picture.

In that spirit, I think that an overall goal here is to maintain enough flexibility in these guidelines to keep them practical with enough rigorous strength to make them a powerful standard. My biggest concern is with the Rules. I do not see how the 3 points outlined are entirely clear and ideally organized. For one thing, I am unsure of how the various subheadings (A, B, C) fit under the headings 1, 2, and 3. For example 1-A and 1-C are mathematical requirements, while 1-B is an implementation issue. Yet, the main headings 1, 2, and 3 are all focused on technical implementation. I would suggest a different organization, perhaps separating mathematical issues from computational/implementation issues. For example how about the following:

Rules for MIASE...
------------------

The overall guiding principle is that reported simulations must be unambiguously reproducible. To do this, it is required that:

1. The mathematical specification of a model must be complete.
	
A. If it is claimed that the model/simulation has a unique solution, then all governing equations, parameter values, and necessary conditions (initial conditions and/or boundary conditions) are provided.

B. For models that include probabilistic/stochastic elements (e.g., chemical master equation, Langevin equation, etc.), then initial state as well as all equations/rules used to generate a trajectory are provided.

C. ...

2. All relevant information on the computational implementation must be provided

A. The simulation algorithm needed to perform the experiment must be clearly identified.

B. ....

------------------

I think that all or nearly all of your points could be organized in this manner, if you think this makes any sense. So, what do you all think? 

Dan



-----Original Message-----
From: Dagmar Köhn [mailto:da...@eb...] 
Sent: Friday, May 29, 2009 1:04 AM
To: Poul Nielsen
Cc: mia...@li...; p.n...@au...; p.h...@au...; Vijayalakshmi Chelliah; j.l...@au...; hsauro@u.washington.edu; ped...@ma...; Catherine Lloyd; e.c...@au...; bza...@in...; ala...@dp...; Beard, Dan; Stefan Hoops
Subject: Re: Dagmar Köhn: MIASE comments

Dear Poul,

having looked at your suggested changes, there is one thing remaining where I tend not to agree. It concerns the use of repeat/reproduce. It is probably a minor thing, but I just wanted to inform you in case you strongly disagree on the first point and want to object.

The last comment I have found on the topic in my Mail folder was by Stefan saying that:

"We need to distinguish between the use of MIASE and its scope. The scope of MIASE is to specify repeatable simulations. The use is to determine reproducibility of scientific results."

You suggested 3 changes of "repeat" into "reproduce" in the document, those were the following:

1) (Information on the Models) The MIASE guidelines demand that those changes are defined within the experiment description in a way that allows their later repetition (i.e. be traceable), leading to the same instance of the model.

Here, I'd say that we are talking about the scope of MIASE rather than the use? I think 'repeat' fits better, so I'd like to leave it.

2) (Discussion) A quantitative model is only useful when it can be simulated in a meaningful way - and in order to do so, model users must be supported in setting up and repeating simulation experiments. Once simulations can be easily reproduced, scientists will start to use models and their simulations as given. The simulations become reliable sources for the composition of models of larger systems which can be re-used instead of starting from the very beginning over and over again.

Here, I agree to change 'repeat' to 'reproduce', i.e.:
A quantitative model is only useful when it can be simulated in a meaningful way - and in order to do so, model users must be supported in setting up and reproducing simulation experiments. Once simulations can be easily reproduced, scientists will start to use models and their simulations as given. The simulations become reliable sources for the composition of models of larger systems which can be re-used instead of starting from the very beginning over and over again.

Best,
Dagmar



Poul Nielsen wrote:
> Dear Dagmar
>
> I have suggested a couple of minor changes to the MIASE document. 
> These should be visible as I have turned 'Track Changes' on, as 
> suggested. It is not clear to me how the altered document should be 
> uploaded so I am sending my version directly to you.
>
> Best wishes
> Poul