dl-learner-discussion

[DL-Learner discussion] Question about calculating accuracy in dllearner

[June <zhu...@gm...>]

Hi, Jens,

I am a little confused about how the accuracy is measured in your paper 
"concept learning in DLs using refinement operator(machine learning 
journal)".

In the API dllearner-1.0-beta-2, suppose 10 solutions can be found with 
100% accuracy and f-measure for the definition of each concept, and 
together there are 5 concept. How did you calculate your accuracy 
measure in the experiment?

Did you calculate the correct definitions manually in the total 50 
solutions, and then divide by 50? Or select at most one correct 
definition from the solutions for one concept, and calculate the count 
divide by 5? Or else?

Sorry for asking such a trivial question~

best regards,

June

Re: [DL-Learner discussion] Question about calculating accuracy in dllearner

[Jens L. <le...@in...>]

Hello June,

Am 20.09.2012 14:44, schrieb June:
> Hi, Jens,
>
> I am a little confused about how the accuracy is measured in your paper
> "concept learning in DLs using refinement operator(machine learning
> journal)".
>
> In the API dllearner-1.0-beta-2, suppose 10 solutions can be found with
> 100% accuracy and f-measure for the definition of each concept, and
> together there are 5 concept. How did you calculate your accuracy
> measure in the experiment?
>
> Did you calculate the correct definitions manually in the total 50
> solutions, and then divide by 50? Or select at most one correct
> definition from the solutions for one concept, and calculate the count
> divide by 5? Or else?

In the experiments in the paper [1], we used cross validation [2]. For 
each fold in the validation, we measured (predictive) accuracy [3], i.e. 
the percentage of correctly classified examples (positives and 
negatives). For this, we always only picked the single best solution 
generated by an algorithm. The DL-Learner commandline interface usually 
displays several solutions, but for the experiments only the first one 
is taken into account. I hope this clarifies the issue.

Kind regards,

Jens

[1] http://jens-lehmann.org/files/2010/concept_learning_mlj.pdf
[2] http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29
[3] 
http://en.wikipedia.org/wiki/Accuracy_and_precision#In_binary_classification

-- 
Dr. Jens Lehmann
Head of AKSW/MOLE group, University of Leipzig
Homepage: http://www.jens-lehmann.org
GPG Key: http://jens-lehmann.org/jens_lehmann.asc