When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search
This addresses the issue of structured variability in LLM-based retrieval for policy-relevant search tasks, which is incremental as it builds on existing concerns about model disagreement.
The study tackled the problem of labeling disagreement between LLMs (LLaMA and Qwen) in relevance filtering for SDG search, finding that disagreement is systematic with consistent lexical patterns and leads to divergent top-ranked outputs, detectable with AUCs above 0.74.
Large language models (LLMs) are increasingly used to assign document relevance labels in information retrieval pipelines, especially in domains lacking human-labeled data. However, different models often disagree on borderline cases, raising concerns about how such disagreement affects downstream retrieval. This study examines labeling disagreement between two open-weight LLMs, LLaMA and Qwen, on a corpus of scholarly abstracts related to Sustainable Development Goals (SDGs) 1, 3, and 7. We isolate disagreement subsets and examine their lexical properties, rank-order behavior, and classification predictability. Our results show that model disagreement is systematic, not random: disagreement cases exhibit consistent lexical patterns, produce divergent top-ranked outputs under shared scoring functions, and are distinguishable with AUCs above 0.74 using simple classifiers. These findings suggest that LLM-based filtering introduces structured variability in document retrieval, even under controlled prompting and shared ranking logic. We propose using classification disagreement as an object of analysis in retrieval evaluation, particularly in policy-relevant or thematic search tasks.