On the Robustness of Generative Retrieval Models: An Out-of-Distribution Perspective
This addresses the robustness issue for the information retrieval community, as generative retrieval models are gaining attention but lack analysis on how they generalize to new distributions, which is critical for real-world applications.
The paper tackles the problem of out-of-distribution (OOD) robustness in generative retrieval models, which generate document identifiers for retrieval, by defining OOD robustness from three perspectives and conducting empirical studies that show these models require enhancement in robustness compared to dense retrieval models.
Recently, we have witnessed generative retrieval increasingly gaining attention in the information retrieval (IR) field, which retrieves documents by directly generating their identifiers. So far, much effort has been devoted to developing effective generative retrieval models. There has been less attention paid to the robustness perspective. When a new retrieval paradigm enters into the real-world application, it is also critical to measure the out-of-distribution (OOD) generalization, i.e., how would generative retrieval models generalize to new distributions. To answer this question, firstly, we define OOD robustness from three perspectives in retrieval problems: 1) The query variations; 2) The unforeseen query types; and 3) The unforeseen tasks. Based on this taxonomy, we conduct empirical studies to analyze the OOD robustness of several representative generative retrieval models against dense retrieval models. The empirical results indicate that the OOD robustness of generative retrieval models requires enhancement. We hope studying the OOD robustness of generative retrieval models would be advantageous to the IR community.