Re: [Miase-discuss] MIASE paper - reproducibility

[Stefan H. <sh...@vb...>]

Hello All,

On Mon, 11 May 2009 09:22:38 +0100
Nicolas Le novère <le...@eb...> wrote:

> Dagmar Köhn wrote:
> 
> > Regarding the reproducibility...
> > Well, being a bit picky, and because we had kind of a related
> > discussion with Frank and Nicolas, I checked the term
> > reproducibility, which is often defined as: "the ability of a test
> > or experiment to be accurately reproduced, or replicated, by
> > someone else working independently." Re-occurring in all the
> > definitions is the fact that in order to be reproducible, an
> > experiment has to be run by several groups/ under different lab
> > conditions. I am not sure, but tend to think that "someone else"
> > might not only include another user on the same simulation tool,
> > but also a different simulation/computational environment?!
> 
> There is nothing in your definition that says the tools have to be 
> different. The investigators must be different. All this is not at
> all specific to computational biology. Experimental science have
> something called materials and methods in papers, that allow someone
> using the same materials, and the same methods, to produce the same
> results and come to the same conclusion. If one finally finds the
> Higg boson using the LHC, the experiment will have to be reproduced
> many times. I am not certain, another multi-billion accelerator will
> be built for that.
> 
> > Maybe we want to avoid the discussion on reproducibility and just
> > state that the repetition of experiments is enabled, which is
> > easier to achieve as repeatable does not demand different
> > conditions per definition. ("Repeatability is the variation in
> > measurements taken by a single person or instrument on the same
> > item and under the same conditions. A measurement may be said to be
> > repeatable when this variation is smaller than some agreed limit.")
> 
> I really believe people won't understand the difference between 
> reproducibility and repeatability (I am not certain I do). Regarding
> the use of different tools, two situations are clear (and were
> discussed in Auckland):
> 1) The simulation description is encoded in a standard language, and
> that language is supported by several simulation tools. No problem.
> 2) The simulation tool is a black-box. In that case, the simulation 
> description should describe exactly how to use the black-box, but
> also be sufficient to permit to run the simulation on another tool,
> including a black-box.
> 
> The grey area is the simulation that is run with an open code (not a 
> black-box), but cannot be encoded in a standard format. What do-we 
> require/request/advise/suggest here? My personal inclination is to
> merge it with the case 1) above. Since the code is open, people can
> reproduce the simulation (yes, it would take a lot of work, time,
> energy and money. But I am not sure we can use that argument at all).
> 
> So at the end we would have open codes and black-boxes. In the first
> case, we request the use of standard formats when a relevant standard
> exist. In the second case, we ask for reproducibility.
> 

I agree with Nicolas that we should ask for reproducibility. Repeating
a simulation is insufficient. If a simulation experiment which is
described in a MIASE compliant format is allowed to have different
results we have not specified anything at least in my opinion. If the
result is not reproducible something is wrong with the initial
experiment, the second experiment or the MIASE compliant description.
Either of the 3 cases can be resolved through further investigation,
which is the normal scientific process.

Thanks,
Stefan
  

-- 
Stefan Hoops, Ph.D.
Senior Project Associate
Virginia Bioinformatics Institute - 0477
Virginia Tech
Bioinformatics Facility II
Blacksburg, Va 24061, USA

Phone: (540) 231-1799
Fax:   (540) 231-2606
Email: sh...@vb...