Data Requirements for Evaluation of Personalization of Information Retrieval - A Position Paper
This is an incremental position paper that identifies key data requirements for evaluating personalized IR systems, targeting researchers and practitioners in information retrieval.
The paper addresses the overlooked need for evaluating personalization in information retrieval across entire search sessions and diverse user activities, proposing necessary data types for meaningful assessment based on interactive IR studies.
Two key, but usually ignored, issues for the evaluation of methods of personalization for information retrieval are: that such evaluation must be of a search session as a whole; and, that people, during the course of an information search session, engage in a variety of activities, intended to accomplish differ- ent goals or intentions. Taking serious account of these factors has major impli- cations for not only evaluation methods and metrics, but also for the nature of the data that is necessary both for understanding and modeling information search, and for evaluation of personalized support for information retrieval (IR). In this position paper, we: present a model of IR demonstrating why these fac- tors are important; identify some implications of accepting their validity; and, on the basis of a series of studies in interactive IR, identify some types of data concerning searcher and system behavior that we claim are, at least, necessary, if not necessarily sufficient, for meaningful evaluation of personalization of IR.