A Formal Account of Effectiveness Evaluation and Ranking Fusion
This work provides a theoretical foundation for information retrieval evaluation, addressing a domain-specific problem for researchers and practitioners in retrieval systems.
The paper tackles the problem of evaluating retrieval system effectiveness and ranking fusion by proposing an information-theoretic framework, resulting in a metric that improves on popular ones in formal constraints and captures quality aspects from traditional metrics.
This paper proposes a theoretical framework which models the information provided by retrieval systems in terms of Information Theory. The proposed framework allows to formalize: (i) system effectiveness as an information theoretic similarity between system outputs and human assessments, and (ii) ranking fusion as an information quantity measure. As a result, the proposed effectiveness metric improves popular metrics in terms of formal constraints. In addition, our empirical experiments suggest that it captures quality aspects from traditional metrics, while the reverse is not true. Our work also advances the understanding of theoretical foundations of the empirically known phenomenon of effectiveness increase when combining retrieval system outputs in an unsupervised manner.