More Room for Language: Investigating the Effect of Retrieval on Language Models
This work addresses the impact of retrieval on language models for NLP researchers, but it is incremental as it focuses on controlled analysis rather than new applications.
The paper investigated how retrieval augmentation affects language models by introducing an 'ideal retrieval' methodology, finding that these models save less world knowledge in their weights, improve at understanding local context and inter-word dependencies, but worsen at comprehending global context.
Retrieval-augmented language models pose a promising alternative to standard language modeling. During pretraining, these models search in a corpus of documents for contextually relevant information that could aid the language modeling objective. We introduce an 'ideal retrieval' methodology to study these models in a fully controllable setting. We conduct an extensive evaluation to examine how retrieval augmentation affects the behavior of the underlying language model. Among other things, we observe that these models: i) save substantially less world knowledge in their weights, ii) are better at understanding local context and inter-word dependencies, but iii) are worse at comprehending global context.