About cross-validation results

[Guangyuan Piao]

I used the -kcv option to get the result of a ranker.

Except each result for each folder, there's a final result such as:
Total: 0.9975 0.08 0.9975

What's the meaning of the three outputs?

[Lemur Project]

This result does not look familiar to me for saved model text.

Did you get any information at the top of the output file that listed algorithm name and settings? What were they?

What version of RankLib are you using and which particular algorithm did you select?

The latest RankLib version is 2.5.

[Jose M]

I have the same question about the meaning of the outputs.

[Lemur Project]

What precisely did you run?

[Jose M]

For example,

java -jar bin/RankLib.jar -feature myfeaturesFile -train train.txt -test test.txt -ranker 0 -metric2t ERR@10 -metric2T RR@10 -norm zscore -gmax 1 -kcv 3

The output is:

---------------------------------
Finished sucessfully.
ERR@10 on training data: 0.3567
---------------------------------
Total: 0.2904   0.3433  0.6128  

What is the meaning of these last three outputs??

Thanks!

[Lemur Project]

Again, what version of RankLib are you using?

For the latest snapshot (RankLib-2.7-SNAPSHOT) the output I obtained using

    java -jar RankLib.jar -feature feature.list -train train.txt 
              -ranker 0  -metric2t ERR@10 -metric2T RR@10 -norm zscore -kcv 3

--------------------------------
Finished sucessfully.
ERR@10 on training data: 0.116
---------------------------------
Summary:
RR@10   |   Train       | Test
----------------------------------
Fold 1  |   0.1181      |  0.4566
Fold 2  |   0.1175      |  0.4611
Fold 3  |   0.116       |  0.4544
----------------------------------
Avg.    |   0.1172      |  0.4574
----------------------------------
Total   |               |  0.4574

This output represents the average ERR@10 for the training runs and the RR@10 for
the test runs over the three folds.

Only the average for the test runs is shown under totals (which is a bit misleading).

I used a feature file of only 8 out of a possible 44 features in my test data.

If you are not using a current version of RankLib, perhaps the three items represent those averages over each of the folds? Do you get four values if you use four folds?

[Jose M]

When I use CV with 5 folds (-kcv 5), shows three values again:

Finished sucessfully.
ERR@10 on training data: 0.349
---------------------------------
Total: 0.4194   0.5426  1.0

[Lemur Project]

Have you made the run using the latest RankLib snapshot?
You still haven't mentioned what version you are using.