Beyond Cumulated Gain and Average Precision: Including Willingness and Expectation in the User Model
This work addresses the need for more accurate user-centric evaluation in information retrieval, though it appears incremental as it builds on existing concepts in the literature.
The paper tackles the problem of evaluating search results by proposing a new metric family that integrates user willingness and expectation into a stopping criterion and satisfaction model, aiming to improve upon existing metrics like cumulated gain and average precision.
In this paper, we define a new metric family based on two concepts: The definition of the stopping criterion and the notion of satisfaction, where the former depends on the willingness and expectation of a user exploring search results. Both concepts have been discussed so far in the IR literature, but we argue in this paper that defining a proper single valued metric depends on merging them into a single conceptual framework.