Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation
This work provides insights for improving training efficiency in retrieval-augmented language models, though it appears incremental as it focuses on understanding existing methods rather than introducing new ones.
The paper investigates attention distillation in retrieval-augmented generation, analyzing its mechanisms and identifying key factors that influence training quality, while proposing optimization indicators to avoid ineffective training.
Retrieval-augmented generation framework can address the limitations of large language models by enabling real-time knowledge updates for more accurate answers. An efficient way in the training phase of retrieval-augmented models is attention distillation, which uses attention scores as a supervision signal instead of manually annotated query-document pairs. Despite its growing popularity, the detailed mechanisms behind the success of attention distillation remain unexplored, particularly the specific patterns it leverages to benefit training. In this paper, we address this gap by conducting a comprehensive review of attention distillation workflow and identifying key factors influencing the learning quality of retrieval-augmented language models. We further propose indicators for optimizing models' training methods and avoiding ineffective training.