Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation
For practitioners building cold-start recommender systems, the paper demonstrates that retrieval bottlenecks, not reranking quality, are the primary limitation, and that current LLM-based reranking pipelines fail to leverage LLMs' potential advantages.
The paper tests the assumption that LLM rerankers improve cold-start recommendation, finding that calibrated LLMs fail to consistently outperform strong baselines even when the gold item is guaranteed present. In realistic retrieval settings, standard retrievers place the gold item in a 200-item pool only 4.6-22.9% of the time, and while a learned hybrid fusion layer (LHF) recovers 17-61% of oracle coverage on content-rich domains, LLM reranking often degrades performance, revealing that LLMs' semantic advantages are largely unreachable in current pipelines.
Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes. We test this assumption with a five-domain benchmark that explicitly separates reranking quality from retrieval coverage. In a positive-controlled regime where the gold item is guaranteed present, calibrated LLM rerankers fail to consistently outperform strong collaborative and content baselines under natural traffic, and within-family scaling from Qwen3-8B to Qwen3-32B narrows but does not close the gap on most domains. In a retrieval-realistic regime where the gold item is not injected, the bottleneck is more severe: standard single retrievers place the gold item in a 200-item pool only 4.6-22.9% of the time, largely because 32-91% of cold-start targets are brand-new items with no training interactions. We introduce LHF, a validation-trained learned hybrid fusion layer over a multi-retriever union pool, as a retrieval-side realizability baseline. LHF is the only combiner we test that beats every single retriever on all five domains and recovers 17-61% of oracle coverage headroom on content-rich domains, but only 5-7% on collaboratively strong domains. End-to-end experiments reveal the remaining mismatch: learned non-LLM ranking exploits the LHF pool, while prompt-level LLM reranking often degrades it. LLMs exhibit pockets of semantic cold-start advantage, especially in text-rich domains when the item is already present, but this advantage is largely unreachable in current retrieve-then-rerank pipelines. We release the benchmark protocol, splits, prompts, evaluation tooling, and archived reproducibility artifacts: data at https://doi.org/10.5281/zenodo.20991039 and code at https://doi.org/10.5281/zenodo.20993306.