Re: [Mulan-list] leave-study-out-cross validation (LSO-CV)
Brought to you by:
stevelaskaridis,
tsoumakas
From: Nadav S. <ns...@gm...> - 2015-05-27 07:38:01
|
Hi Greg, thanks for your answer. I'll clerify my question- In my data each sample has a dataset_ID that tells me to which dataset it belongs (All of the datasets have the same features and labels). Instead of cross validating by splitting the data into k folds (i.e. on each iteration excluding one fold while training and then predicting that fold), I'd like to to split the data to the data sets mentioned (using their dataset ID) and then on each iteration- - Exclude one of the data sets (i.e. exclude all samples that belong to that data set) and train the model on the remaining data. - Generate prediction for the data set left out. In the end I'd like to compare the predicted matrix to the original one and get all relevant performance measurements. This is essentially the same as cross validating with k-folds, only that now each fold is a different data set. Any suggestion on how to do so with Mulan? Also, just to be sure- what's exactly the return value of the evaluate function? Thank you, Nadav. 2015-05-26 23:52 GMT+03:00 Grigorios Tsoumakas <gr...@cs...>: > Hi, > > I have about 20 data sets, and I'd like to perform a > leave-study-out-cross validation, i.e. on each iteration I want to train > the model on all data sets but one, and then cross validate using the one > data set left out. > Is there a built-in way to do that in Mulan? > > > I guess you mean some kind of meta-learning model or hyper-parameter > optimization approach, otherwise leave-study-out does not really make > sense. In any case, Mulan does not support this. > > If not, should I use the Evaluator.evaluate function in order to cross > validate the one data set left out? > > > I can't say I understand this question. In > leave-study-out-cross-validation, you should be learning on the 19 datasets > and predicting on the 20th, why would you want to do cross-validation on > the 20th dataset? I suppose you mean to evaluate a model on the 20th > dataset. In this case, Evaluator.evaluate function would do the trick. > > Hope this helps, > Greg > > > > On 26/5/2015 5:52 μμ, Nadav Stiner wrote: > > Hi everyone, > > I have about 20 data sets, and I'd like to perform a > leave-study-out-cross validation, i.e. on each iteration I want to train > the model on all data sets but one, and then cross validate using the one > data set left out. > Is there a built-in way to do that in Mulan? > If not > , should I use the Evaluator.evaluate function in order to cross validate > the one data set left out? > > Thanks! > Nadav. > > > ------------------------------------------------------------------------------ > One dashboard for servers and applications across Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable Insights > Deep dive visibility with transaction tracing using APM Insight.http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > > > > _______________________________________________ > Mulan-list mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/mulan-list > > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Mulan-list mailing list > Mul...@li... > https://lists.sourceforge.net/lists/listinfo/mulan-list > > |