#1 matthews correlation coefficient (mcc) et al.

[ozi]

matthews correlation coefficient (mcc) is a valuable
tool for model selection.

Especially if classes are very unevenly distributed you
can get acceptable values for ROC, ppv, balancedSuccess
etc. although the classifier assigns all examples to
the same class. mcc gives 0 in such cases. AFAIK the
extension to more than two classes ist straightforward
(i think the Baldi/Brunak book has a formula).

However pure mcc assigns 0 also to perfect classifiers,
so one needs probably a measure which is mcc derived.
The minimum and the standard deviation of the
performance measures obtained by stratifiedCV are also
important indicators and should be given together with
the average.

In summary I would like to see a bit more flexibility
in how the best model is selected from e.g. a ParamGrid.

Thank you.

[abh]

Logged In: YES
user_id=1033568

model selection now supports selection according to other performance
metrics.