A Usefulness-based Approach for Measuring the Local and Global Effect of IIR Services
This work addresses the evaluation challenges for IIR services, offering a log-based method to assess impact beyond traditional metrics, though it is incremental in applying an existing model to a specific domain.
The authors tackled the problem of evaluating interactive information retrieval services by operationalizing a usefulness model to measure both local and global effects on the search process, finding that using a search term suggestion service significantly increases positive signals in subsequent session steps.
In Interactive Information Retrieval (IIR) different services such as search term suggestion can support users in their search process. The applicability and performance of such services is either measured with different user-centered studies (like usability tests or laboratory experiments) or, in the context of IR, with their contribution to measures like precision and recall. However, each evaluation methodology has its certain disadvantages. For example, user-centered experiments are often costly and small-scaled; IR experiments rely on relevance assessments and measure only relevance of documents. In this work we operationalize the usefulness model of Cole et al. (2009) on the level of system support to measure not only the local effect of an IR service, but the impact it has on the whole search process. We therefore use a log-based evaluation approach which models user interactions within sessions with positive signals and apply it for the case of a search term suggestion service. We found that the usage of the service significantly often implicates the occurrence of positive signals during the following session steps.