Attentive Pooling Networks
This work addresses the challenge of improving neural network models for pair-wise tasks like question answering, offering a general framework that enhances representation learning, though it is incremental as it builds on existing attention mechanisms.
The authors tackled the problem of discriminative model training for pair-wise ranking or classification by proposing Attentive Pooling (AP), a two-way attention mechanism that allows pooling layers to incorporate mutual influence between input pairs, resulting in state-of-the-art performance across three benchmark tasks in question answering/answer selection.
In this work, we propose Attentive Pooling (AP), a two-way attention mechanism for discriminative model training. In the context of pair-wise ranking or classification with neural networks, AP enables the pooling layer to be aware of the current input pair, in a way that information from the two input items can directly influence the computation of each other's representations. Along with such representations of the paired inputs, AP jointly learns a similarity measure over projected segments (e.g. trigrams) of the pair, and subsequently, derives the corresponding attention vector for each input to guide the pooling. Our two-way attention mechanism is a general framework independent of the underlying representation learning, and it has been applied to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) in our studies. The empirical results, from three very different benchmark tasks of question answering/answer selection, demonstrate that our proposed models outperform a variety of strong baselines and achieve state-of-the-art performance in all the benchmarks.