Re: [Miase-discuss] MIASE paper - reproducibility

[Dagmar K. <da...@eb...>]

I know I should let go, but I can't :-)
Those things are just not clear to me.
I'll try to put simple theses and hopefully can *explain* how I was 
interpreting comments.
Please, correct me.


I agree that Stefan got Nicolas wrong, who was advocating for repeatability.
I agree with Stefan that repetition is not sufficient.
I also agree that "if a simulation experiment which is described in a 
MIASE compliant format is allowed to have different results  we have not 
specified anything". (in the sense that we have not specified the MIASE 
rules appropriately)
I think that Nicolas interpreted "If a simulation experiment which is 
described in a MIASE compliant format is allowed to have different 
results we have not specified anything at least in my opinion." as 
Stefan demanding to repeat a simulation experiment on different 
simulators in order to judge it MIASE-compliant (and associated that 
with the term "correct").
If that was indeed Stefan's reason for reproducibility, I disagree with him.

The reason why I would like to see reproducibility enabled is that I 
would be very happy if a MIASE compliant simulation experiment 
description was general enough to be applied to more than exactly the 
same simulation tool simulating exactly the same model and running on 
exactly the same machine with exactly the same algorithm/integrator and 
so forth ...
I don't need a MIASE-specification for correctness of simulation 
results, but for RE-USABILITY (and I don't think "correctness" should be 
part of this discussion about repeatability/reproducibility at all).

Therefore, I'd like to have MIASE rules set up in a way that enable 
reproducibility rather than repeatability only.

That must not hold for all simulations, but should be possible in general.

Dagmar

PS: Stefan, I'd say that for the purpose "to determine reproducibility 
of scientific results" you would rather USE the SED-ML than MIASE, so if 
the question is how to use SED-ML to verify simulation results, then it 
should imho be discussed elsewhere.


Stefan Hoops wrote:
> Hello Nicolas,
>
> On Mon, 11 May 2009 16:07:34 +0100 (BST)
> "Nicolas Le Novere" <le...@eb...> wrote:
>
>   
>>> I agree with Nicolas that we should ask for reproducibility.
>>>       
>> I think you got me wrong. I actually advocated the opposite. And I
>> believed I was reflecting the Auckland consensus, that said MIASE did
>> not deal with correctness.
>>
>>     
>>> Repeating
>>> a simulation is insufficient. If a simulation experiment which is
>>> described in a MIASE compliant format is allowed to have different
>>> results we have not specified anything at least in my opinion.Yes, we specified a way to discover the discrepancy, which is exactly
>>> what a materials and methods is for. Results are described in another
>>> part of the paper, called ... results. We should encourage the use of
>>> standard formats to describe numerical results, but I do not believe
>>> the results themselves are part of a "simulation experiment
>>> description". And even less the correctness criteria to compare two
>>> sets of results.
>>>
>>>       
>
> I think we actually agree, I was not precise enough. We need to
> distinguish between the use of MIASE and its scope. The scope of MIASE
> is to specify repeatable simulations. The use is to determine
> reproducibility of scientific results.
>
> Thanks,
> Stefan
>
>
>