Hi,
I have a rather philosophic question.
I noticed that there is a problem with NDCG measure when the list of relevance labels consists only from zeros. It means for example
list_of_labels = [0,0,0,0,0,0,0,0,0,0]
where the array list_of_labels is an array of relevance labels of all 10 documents from the given query.
What happens then is, that the ideal DCG equals zero.
IDCG(list_of_labels) = 0
My question is: Why have you decided to define NDCG(list_of_labels) = 0? Is it the way how it is defined in literature? Or do you believe it is a correct way? Or do you think that it doesnt matter and you just assigned zero because you considered it suitable?
Wouldn't it be more reasonable to just skip such a list of labels and leave it out of the scoring? As it doesn't provide us with any useful information? I don't think that the zero would be really a mistake or that the NDCG score would be incorrect, I am just curious about your opinion on this topic because of my own implementation of NDCG in Python.
Thank you!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
My question is: Why have you decided to define NDCG(list_of_labels) = 0? Is it the
way how it is defined in literature? Or do you believe it is a correct way? Or do
you think that it doesnt matter and you just assigned zero because you considered
it suitable?
I chose this implementation because it's intuitive to me (i.e., you can never do well on queries that don't have any relevant documents, thus the "goodness" of your ranking is always 0).
Wouldn't it be more reasonable to just skip such a list of labels and leave it out
of the scoring?
Yes, I think it is. In RankLib, you can use "-hr" to do exactly this :-) However, in my experiments, I didn't notice any meaningful difference between skipping such lists and leaving them in.
Last edit: Van Dang 2015-01-13
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I have a rather philosophic question.
I noticed that there is a problem with NDCG measure when the list of relevance labels consists only from zeros. It means for example
list_of_labels = [0,0,0,0,0,0,0,0,0,0]
where the array list_of_labels is an array of relevance labels of all 10 documents from the given query.
What happens then is, that the ideal DCG equals zero.
IDCG(list_of_labels) = 0
My question is: Why have you decided to define NDCG(list_of_labels) = 0? Is it the way how it is defined in literature? Or do you believe it is a correct way? Or do you think that it doesnt matter and you just assigned zero because you considered it suitable?
Wouldn't it be more reasonable to just skip such a list of labels and leave it out of the scoring? As it doesn't provide us with any useful information? I don't think that the zero would be really a mistake or that the NDCG score would be incorrect, I am just curious about your opinion on this topic because of my own implementation of NDCG in Python.
Thank you!
Better late than never.
I chose this implementation because it's intuitive to me (i.e., you can never do well on queries that don't have any relevant documents, thus the "goodness" of your ranking is always 0).
Yes, I think it is. In RankLib, you can use "-hr" to do exactly this :-) However, in my experiments, I didn't notice any meaningful difference between skipping such lists and leaving them in.
Last edit: Van Dang 2015-01-13