Re: [Miase-discuss] MIASE paper - reproducibility

[Nicolas Le n. <le...@eb...>]

Dagmar Köhn wrote:

> Regarding the reproducibility...
> Well, being a bit picky, and because we had kind of a related discussion 
> with Frank and Nicolas, I checked the term reproducibility, which is 
> often defined as: "the ability of a test or experiment to be accurately 
> reproduced, or replicated, by someone else working independently." 
> Re-occurring in all the definitions is the fact that in order to be 
> reproducible, an experiment has to be run by several groups/ under 
> different lab conditions.
> I am not sure, but tend to think that "someone else" might not only 
> include another user on the same simulation tool, but also a different 
> simulation/computational environment?!

There is nothing in your definition that says the tools have to be 
different. The investigators must be different. All this is not at all 
specific to computational biology. Experimental science have something 
called materials and methods in papers, that allow someone using the same 
materials, and the same methods, to produce the same results and come to 
the same conclusion. If one finally finds the Higg boson using the LHC, the 
experiment will have to be reproduced many times. I am not certain, another 
multi-billion accelerator will be built for that.

> Maybe we want to avoid the discussion on reproducibility and just state 
> that the repetition of experiments is enabled, which is easier to 
> achieve as repeatable does not demand different conditions per 
> definition. ("Repeatability is the variation in measurements taken by a 
> single person or instrument on the same item and under the same 
> conditions. A measurement may be said to be repeatable when this 
> variation is smaller than some agreed limit.")

I really believe people won't understand the difference between 
reproducibility and repeatability (I am not certain I do). Regarding the 
use of different tools, two situations are clear (and were discussed in 
Auckland):
1) The simulation description is encoded in a standard language, and that 
language is supported by several simulation tools. No problem.
2) The simulation tool is a black-box. In that case, the simulation 
description should describe exactly how to use the black-box, but also be 
sufficient to permit to run the simulation on another tool, including a 
black-box.

The grey area is the simulation that is run with an open code (not a 
black-box), but cannot be encoded in a standard format. What do-we 
require/request/advise/suggest here? My personal inclination is to merge it 
with the case 1) above. Since the code is open, people can reproduce the 
simulation (yes, it would take a lot of work, time, energy and money. But I 
am not sure we can use that argument at all).

So at the end we would have open codes and black-boxes. In the first case, 
we request the use of standard formats when a relevant standard exist. In 
the second case, we ask for reproducibility.

-- 
Nicolas LE NOVERE, Computational Neurobiology, EMBL-EBI, Wellcome-Trust
Genome Campus, Hinxton CB101SD UK, Mob:+447833147074, Tel:+441223494521
Fax:468,Skype:n.lenovere,AIM:nlenovere,MSN:nle...@ho...(NOT email)
http://www.ebi.ac.uk/~lenov/, http://www.ebi.ac.uk/compneur/