Document Distance for the Automated Expansion of Relevance Judgements for Information Retrieval Evaluation
This work addresses the challenge of insufficient relevance judgments for researchers in information retrieval, though it is incremental as it builds on existing document distance methods.
The paper tackled the problem of limited relevance judgments in information retrieval evaluation by using a document distance-based approach to automatically expand the number of available judgments, showing that evaluations with expanded judgments are more reliable, especially when initial judgments are very limited, as demonstrated on OHSUMED and TREC-8 datasets.
This paper reports the use of a document distance-based approach to automatically expand the number of available relevance judgements when these are limited and reduced to only positive judgements. This may happen, for example, when the only available judgements are extracted from a list of references in a published review paper. We compare the results on two document sets: OHSUMED, based on medical research publications, and TREC-8, based on news feeds. We show that evaluations based on these expanded relevance judgements are more reliable than those using only the initially available judgements, especially when the number of available judgements is very limited.