Re: [Mulan-list] Mulan: Train a model, validation and testing

[Grigorios T. <gr...@cs...>]

Dear Yong,

What you can do is use the function "build" of your classifier on the 
training set, and then call the method "evaluate" of the evaluator on 
the validation set. See class mulan.examples.TrainTestExperiment for an 
example (test equals validation in this case).

For obtaining predictions on unlabeled data, see class 
mulan.examples.ObtainingPredictionsOnUnlabeledData

hth,
Greg


On 16/9/2015 3:45 πμ, Yong Zheng wrote:
> Hello,
>
> I'd like to have three data sets: training, validation and testing.
>
> Typically, I'd like to train a model on the training set, where I can 
> get the training accuracy, such as accuracy, precision, recall, etc.
>
> After that, I'd like to apply this model to the validation set, since 
> the results based on training set may be overfitted. In this case, I 
> want to get the same metrics on valdiation set.
>
> Finally, by choosing the best model, I can use the testing set.
>
> So, both the training and validatoin set, I may have the real labels, 
> But probably there are no labels on the testing set.
>
> Currently, I used a 10-folds cross validation on the training set to 
> train a model, Next, I'd like to apply this model to the validation 
> set and get some metric values. I noticed there is a cross-validation 
> function, and also a prediction module for providing label 
> predictions, but how to apply the model to the validation set and get 
> the metrics????
>
> In testing, I know, the labels should be marked as the question mark 
> ?, but how about the validation set, where we have real labels. One 
> way I can figure out is to mark the labels as ? in validation set and 
> got label predictions from the model. After that I can manually 
> calculate the metrics. Are there any other simple ways to let the 
> system automatically validate it and generate the metric values?
>
> Thanks,
>
> Yong
>
>
> ------------------------------------------------------------------------------
> Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
> Get real-time metrics from all of your servers, apps and tools
> in one place.
> SourceForge users - Click here to start your Free Trial of Datadog now!
> http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
>
>
> _______________________________________________
> Mulan-list mailing list
> Mul...@li...
> https://lists.sourceforge.net/lists/listinfo/mulan-list