Understanding and Debugging Failures in N-Gram-Based Generative Retrieval
For the information retrieval community, this work provides a systematic understanding and debugging tool for failures in n-gram-based generative retrieval, though it is incremental as it focuses on a specific subset of methods.
The paper presents a taxonomy of failure modes in generative retrieval, empirically investigates failures in n-gram-based methods (SEAL and MINDER), and introduces a web-based tool for analyzing generated n-grams. Key findings include issues like ambiguous docids, low identifier diversity, and disproportionate impact of specific identifiers.
Generative Retrieval (GR) is an emerging Information Retrieval (IR) paradigm that is motivated by increasingly capable language models. In GR, a model directly generates identifiers for relevant documents. While these systems offer unique advantages, they also introduce distinct failure mechanisms. We explore these failure modes in three contributions: (1) We present a taxonomy of GR failure modes based on GR literature. (2) We empirically investigate failure in a subset of GR: ngram-based methods, more specifically, SEAL and MINDER. Our analysis reveals common issues, such as ambiguous docids, low identifier diversity, and the disproportionate impact of specific identifiers. (3) We introduce a new web-based tool that helps the IR community analyze generated ngrams and their respective contribution to the final ranking, providing an intuitive interface to identify where such GR methods go wrong.