Re: [Miase-discuss] MIASE paper - reproducibility

[Andrew M. <ak....@au...>]

Dagmar Köhn wrote:
> I know I should let go, but I can't :-)
> Those things are just not clear to me.
> I'll try to put simple theses and hopefully can *explain* how I was 
> interpreting comments.
> Please, correct me.
>
>
> I agree that Stefan got Nicolas wrong, who was advocating for repeatability.
> I agree with Stefan that repetition is not sufficient.
> I also agree that "if a simulation experiment which is described in a 
> MIASE compliant format is allowed to have different results  we have not 
> specified anything". (in the sense that we have not specified the MIASE 
> rules appropriately)
> I think that Nicolas interpreted "If a simulation experiment which is 
> described in a MIASE compliant format is allowed to have different 
> results we have not specified anything at least in my opinion." as 
> Stefan demanding to repeat a simulation experiment on different 
> simulators in order to judge it MIASE-compliant (and associated that 
> with the term "correct").
> If that was indeed Stefan's reason for reproducibility, I disagree with him.
>
> The reason why I would like to see reproducibility enabled is that I 
> would be very happy if a MIASE compliant simulation experiment 
> description was general enough to be applied to more than exactly the 
> same simulation tool simulating exactly the same model and running on 
> exactly the same machine with exactly the same algorithm/integrator and 
> so forth ...
> I don't need a MIASE-specification for correctness of simulation 
> results, but for RE-USABILITY (and I don't think "correctness" should be 
> part of this discussion about repeatability/reproducibility at all).
>
> Therefore, I'd like to have MIASE rules set up in a way that enable 
> reproducibility rather than repeatability only.
>
> That must not hold for all simulations, but should be possible in general.
>   
Hi,

If models are particularly sensitive to details like the exact rounding 
of floating point numbers, minor algorithmic details, and so on, and the 
author can't explain why we should believe one set of results but not 
the other, then that means that the results of the model should be 
considered unreliable.

When considering how to deal with this, I think we should be careful not 
to grow the scope of MIASE too much; the scope should properly be to 
ensure people disclose enough information about their model to allow the 
scientific process to take place, and not to establish good modelling or 
scientific practices directly.

However, one minimum information item we could add to address this, 
while remaining on scope would be:
   "F. Where the author is aware that the model will produce different 
results in a different simulation environment or, on a different 
computational platform, an explanation of why the model should be run on 
the specified platform in order to properly achieve the purpose of the 
modeling."

Best regards,
Andrew

> Dagmar
>
> PS: Stefan, I'd say that for the purpose "to determine reproducibility 
> of scientific results" you would rather USE the SED-ML than MIASE, so if 
> the question is how to use SED-ML to verify simulation results, then it 
> should imho be discussed elsewhere.
>
>
> Stefan Hoops wrote:
>   
>> Hello Nicolas,
>>
>> On Mon, 11 May 2009 16:07:34 +0100 (BST)
>> "Nicolas Le Novere" <le...@eb...> wrote:
>>
>>   
>>     
>>>> I agree with Nicolas that we should ask for reproducibility.
>>>>       
>>>>         
>>> I think you got me wrong. I actually advocated the opposite. And I
>>> believed I was reflecting the Auckland consensus, that said MIASE did
>>> not deal with correctness.
>>>
>>>     
>>>       
>>>> Repeating
>>>> a simulation is insufficient. If a simulation experiment which is
>>>> described in a MIASE compliant format is allowed to have different
>>>> results we have not specified anything at least in my opinion.Yes, we specified a way to discover the discrepancy, which is exactly
>>>> what a materials and methods is for. Results are described in another
>>>> part of the paper, called ... results. We should encourage the use of
>>>> standard formats to describe numerical results, but I do not believe
>>>> the results themselves are part of a "simulation experiment
>>>> description". And even less the correctness criteria to compare two
>>>> sets of results.
>>>>
>>>>       
>>>>         
>> I think we actually agree, I was not precise enough. We need to
>> distinguish between the use of MIASE and its scope. The scope of MIASE
>> is to specify repeatable simulations. The use is to determine
>> reproducibility of scientific results.
>>
>> Thanks,
>> Stefan
>>
>>
>>   
>>     
>
>
> ------------------------------------------------------------------------------
> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
> production scanning environment may not be a perfect world - but thanks to
> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
> Series Scanner you'll get full speed at 300 dpi even with all image 
> processing features enabled. http://p.sf.net/sfu/kodak-com
> _______________________________________________
> Miase-discuss mailing list
> Mia...@li...
> https://lists.sourceforge.net/lists/listinfo/miase-discuss
>