Enhancing Sentence Embedding with Generalized Pooling
This work addresses sentence representation for NLP applications, but it is incremental as it builds on existing pooling and attention methods.
The paper tackles the problem of improving sentence embeddings by proposing a generalized pooling method using vector-based multi-head attention with penalization terms to reduce redundancy. The result is state-of-the-art performance on four datasets across natural language inference, author profiling, and sentiment classification tasks.
Pooling is an essential component of a wide variety of sentence representation and embedding models. This paper explores generalized pooling methods to enhance sentence embedding. We propose vector-based multi-head attention that includes the widely used max pooling, mean pooling, and scalar self-attention as special cases. The model benefits from properly designed penalization terms to reduce redundancy in multi-head attention. We evaluate the proposed model on three different tasks: natural language inference (NLI), author profiling, and sentiment classification. The experiments show that the proposed model achieves significant improvement over strong sentence-encoding-based methods, resulting in state-of-the-art performances on four datasets. The proposed approach can be easily implemented for more problems than we discuss in this paper.