Evaluating diversity and redundancy-based search metrics independently

Abstract

This paper proposes a new evaluation metric, normalized Coverage Frequency (nCF), which aims to explicitly evaluate the diversity of search results, going beyond the drawbacks of previously proposed measures. In fact, two of the most widely adopted metrics for the diversity retrieval task, namely-nDCG and Intent-Aware Expected Reciprocal Rank (ERR-IA), explicitly evaluate redundancy, but not diversity. While there exists a genuine diversity-based metric called Intent Recall (I-rec), it has some drawbacks. These drawbacks may be inherited by other derived metrics such as D#-nDCG, which combines I-rec with a modified version of nDCG. The proposed nCF metric assesses how often query-intents are successfully covered throughout a ranked list up to a given rank position. A comprehensive study is conducted using both real and synthetic data to compare nCF with-nDCG, ERR-IA, I-recall and D#-nDCG. Results show that the proposed metric correlates well with the existing ones while it is capable of capturing other factors, e.g., a series of coverage. In addition, we categorize the existing metrics into two distinct groups, i.e., diversity and novelty, based upon their intuitive measurements and suggest that they be used independently according to what they quantify for the ease of performance interpretation.