BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives
This work addresses the problem of assessing retrieval systems for complex user needs, particularly for researchers in information retrieval and AI, but it is incremental as it builds on existing benchmarking efforts.
The authors introduced BIRCO, a benchmark for evaluating information retrieval systems on complex, multi-faceted user objectives, and found that no existing approach achieved satisfactory performance across all tasks, indicating a need for improved models and protocols.
We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs.