Re: [Mulan-list] leave-study-out-cross validation (LSO-CV)

[Nadav S. <ns...@gm...>]

Thanks Greg, I appreciate the help.

On יום ו׳, 29 במאי 2015, 12:31 Grigorios Tsoumakas <gr...@cs...> wrote:

>  Hi Nadav,
>
> Ok, now I understand what you want to accomplish. This cannot be done with
> an existing Mulan function. You could do it with your own code and call
> evaluate on the remaining dataset as you correctly thought.
>
> As Mulan is open source, you can see yourself more details on what exactly
> evaluate returns and what is the Boolean array in the MultiLabelOutput
> object that is returned by makePrediction. Briefly, the former is an
> Evaluation object encapsulating results for a list of evaluation measures,
> while the latter corresponds to true/false values indicating relevance or
> not of the specific instance with the corresponding label.
>
> Cheers,
> Greg
>
> On 27/5/2015 10:37 πμ, Nadav Stiner wrote:
>
>  Hi Greg,
>
>  thanks for your answer.
> I'll clerify my question-
> In my data each sample has a dataset_ID that tells me to which dataset it
> belongs (All of the datasets have the same features and labels). Instead of
> cross validating by splitting the data into k folds (i.e. on each iteration
> excluding one fold while training and then predicting that fold), I'd like
> to to split the data to the data sets mentioned (using their dataset ID)
> and then on each iteration-
>
>    - Exclude one of the data sets (i.e. exclude all samples that belong
>    to that data set) and train the model on the remaining data.
>    - Generate prediction for the data set left out.
>
> In the end I'd like to compare the predicted matrix to the original one
> and get all relevant performance measurements.
> This is essentially the same as cross validating with k-folds, only that
> now each fold is a different data set.
>
>  Any suggestion on how to do so with Mulan?
>
>  Also, just to be sure- what's exactly the return value of the evaluate
> function?
>
>  Thank you,
> Nadav.
>
>  2015-05-26 23:52 GMT+03:00 Grigorios Tsoumakas <gr...@cs...>:
>
>>  Hi,
>>
>>  I have about 20 data sets, and I'd like to perform a
>> leave-study-out-cross validation, i.e. on each iteration I want to train
>> the model on all data sets but one, and then cross validate using the one
>> data set left out.
>>  Is there a built-in way to do that in Mulan?
>>
>>
>>  I guess you mean some kind of meta-learning model or hyper-parameter
>> optimization approach, otherwise leave-study-out does not really make
>> sense. In any case, Mulan does not support this.
>>
>> If not​, should I use the Evaluator.evaluate function in order to cross
>> validate the one data set left out?
>>
>>
>> I can't say I understand this question. In
>> leave-study-out-cross-validation, you should be learning on the 19 datasets
>> and predicting on the 20th, why would you want to do cross-validation on
>> the 20th dataset? I suppose you mean to evaluate a model on the 20th
>> dataset. In this case, Evaluator.evaluate function would do the trick.
>>
>> Hope this helps,
>> Greg
>>
>>
>>
>> On 26/5/2015 5:52 μμ, Nadav Stiner wrote:
>>
>>   Hi everyone,
>>
>>  I have about 20 data sets, and I'd like to perform a
>> leave-study-out-cross validation, i.e. on each iteration I want to train
>> the model on all data sets but one, and then cross validate using the one
>> data set left out.
>>  Is there a built-in way to do that in Mulan?
>>  If not
>> ​, should I use the Evaluator.evaluate function in order to cross
>> validate the one data set left out?
>>
>>  Thanks!
>>  Nadav.​
>>
>>
>>  ------------------------------------------------------------------------------
>> One dashboard for servers and applications across Physical-Virtual-Cloud
>> Widest out-of-the-box monitoring support with 50+ applications
>> Performance metrics, stats and reports that give you Actionable Insights
>> Deep dive visibility with transaction tracing using APM Insight.http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
>>
>>
>>
>> _______________________________________________
>> Mulan-list mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/mulan-list
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> Mulan-list mailing list
>> Mul...@li...
>> https://lists.sourceforge.net/lists/listinfo/mulan-list
>>
>>
>
>
> ------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Mulan-list mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/mulan-list
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Mulan-list mailing list
> Mul...@li...
> https://lists.sourceforge.net/lists/listinfo/mulan-list
>