Investigating Order Effects in Multidimensional Relevance Judgment using Query Logs
This work addresses the issue of how sequential decision-making influences relevance judgments in information retrieval, which is incremental as it extends prior user studies to real-world data.
The paper tackled the problem of order effects in multidimensional relevance judgments by proposing a methodology to detect irrational user behavior in large-scale web search query logs, and validated it using a quantum cognitive explanation.
There is a growing body of research which has investigated relevance judgment in IR being influenced by multiple factors or dimensions. At the same time, the Order Effects in sequential decision making have been quantitatively detected and studied in Mathematical Psychology. Combining the two phenomena, there have been some user studies carried out which investigate the Order Effects and thus incompatibility in different dimensions of relevance. In this work, we propose a methodology for carrying out such an investigation in large scale and real world data using query logs of a web search engine, and device a test to detect the presence of an irrational user behavior in relevance judgment of documents. We further validate this behavior through a Quantum Cognitive explanation of the Order and Context effects.