Query-Document Dense Vectors for LLM Relevance Judgment Bias Analysis
This work addresses the reliability of LLMs as relevance assessors in information retrieval, identifying systematic biases to improve evaluation accuracy, though it is incremental as it builds on prior reliability studies.
The paper tackled the problem of systematic mistakes in LLM relevance judgments for IR evaluation by proposing a clustering-based framework to embed query-document pairs into a joint semantic space, revealing that systematic disagreements between humans and LLMs are concentrated in specific semantic clusters, such as definition-seeking or ambiguous contexts, rather than being random.
Large Language Models (LLMs) have been used as relevance assessors for Information Retrieval (IR) evaluation collection creation due to reduced cost and increased scalability as compared to human assessors. While previous research has looked at the reliability of LLMs as compared to human assessors, in this work, we aim to understand if LLMs make systematic mistakes when judging relevance, rather than just understanding how good they are on average. To this aim, we propose a novel representational method for queries and documents that allows us to analyze relevance label distributions and compare LLM and human labels to identify patterns of disagreement and localize systematic areas of disagreement. We introduce a clustering-based framework that embeds query-document (Q-D) pairs into a joint semantic space, treating relevance as a relational property. Experiments on TREC Deep Learning 2019 and 2020 show that systematic disagreement between humans and LLMs is concentrated in specific semantic clusters rather than distributed randomly. Query-level analyses reveal recurring failures, most often in definition-seeking, policy-related, or ambiguous contexts. Queries with large variation in agreement across their clusters emerge as disagreement hotspots, where LLMs tend to under-recall relevant content or over-include irrelevant material. This framework links global diagnostics with localized clustering to uncover hidden weaknesses in LLM judgments, enabling bias-aware and more reliable IR evaluation.