On the Learning Dynamics of Attention Networks
This work addresses the problem of optimizing attention mechanisms in machine learning, offering insights and a hybrid method for researchers and practitioners, though it is incremental in nature.
The paper analyzes the learning dynamics of three attention paradigms (soft, hard, and LVML) and finds that soft attention improves quickly early on but slows later, while hard attention does the opposite. It proposes a hybrid approach that combines their advantages, demonstrating it on semi-synthetic and real-world datasets.
Attention models are typically learned by optimizing one of three standard loss functions that are variously called -- soft attention, hard attention, and latent variable marginal likelihood (LVML) attention. All three paradigms are motivated by the same goal of finding two models -- a `focus' model that `selects' the right \textit{segment} of the input and a `classification' model that processes the selected segment into the target label. However, they differ significantly in the way the selected segments are aggregated, resulting in distinct dynamics and final results. We observe a unique signature of models learned using these paradigms and explain this as a consequence of the evolution of the classification model under gradient descent when the focus model is fixed. We also analyze these paradigms in a simple setting and derive closed-form expressions for the parameter trajectory under gradient flow. With the soft attention loss, the focus model improves quickly at initialization and splutters later on. On the other hand, hard attention loss behaves in the opposite fashion. Based on our observations, we propose a simple hybrid approach that combines the advantages of the different loss functions and demonstrates it on a collection of semi-synthetic and real-world datasets