Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism
This addresses reliability issues in information retrieval for users, but it is incremental as it builds on existing reranking methods.
The paper tackles the problem of frequent failures in neural information retrieval systems by proposing a lightweight abstention mechanism for reranking, demonstrating its efficacy in black-box scenarios with open-source code for replication.
Neural Information Retrieval (NIR) has significantly improved upon heuristic-based Information Retrieval (IR) systems. Yet, failures remain frequent, the models used often being unable to retrieve documents relevant to the user's query. We address this challenge by proposing a lightweight abstention mechanism tailored for real-world constraints, with particular emphasis placed on the reranking phase. We introduce a protocol for evaluating abstention strategies in black-box scenarios (typically encountered when relying on API services), demonstrating their efficacy, and propose a simple yet effective data-driven mechanism. We provide open-source code for experiment replication and abstention implementation, fostering wider adoption and application in diverse contexts.