Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models
This addresses a bottleneck in retriever-reader models for NLP tasks, offering a general regularization strategy to improve performance, though it is incremental as it builds on existing model frameworks.
The paper tackles the problem of retriever-reader models overfitting to top-ranked retrieval passages, which hinders reasoning over the entire set, by introducing a learnable passage mask mechanism that desensitizes top passages and prevents overfitting. The method consistently outperforms baselines across tasks like open question answering, dialogue conversation, and fact verification, as shown in experiments.
Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations. In this work, we notice these models easily overfit the top-rank retrieval passages and standard training fails to reason over the entire retrieval passages. We introduce a learnable passage mask mechanism which desensitizes the impact from the top-rank retrieval passages and prevents the model from overfitting. Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages. Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.